SET

We've started a new initiative here in Stockholm - a discussion group for us "Exploratory Testers", providing a forum for exchange of ideas and war stories.

The last edition featured a dear colleague of mine who told the story of  Test Fests past. Two tales, in fact, where the first test fest was held over a weekend, featuring both programmers and testers, over a stock exchange system. The second invited end-users at the Swedish parliament.

An intriguing topic, it seemed, and the small-ish but brave congregation bombarded the raconteur with questions for well over the hour.

Most appeared to leave the forum invigorated and energized, with fresh inspiration for hosting test fests of their own.

Get in touch with us through our LinkedIn group if you would like to participate in the future, and visit the same group for discussions after the actual events. We expect the next SET forum to take place sometime this May.

No Computer?

After attending one of Gojko Adzic's inspirational seminars, The Chief sent an e-mail to all testers in the office. A key point at the seminar was that "testers shouldn't be testing - they should teach testing and help identifying risks with the developers while designing, etc". The e-mail contained a single, though-provoking, question: "Could you do an entire sprint in your teams and maintain quality – without computers?"


I wrote the following reply, which relates to the role I'm currently in, but I'm curious about how this works in other places;




"First off, what do we do during a sprint?

· Release testing – checking core functionality while keeping our eyes open for regressions in a soon-to-be-released version
· Improving the automated test harness – Jbehave tests, etc, to minimize the amount of future manual regression testing
· Testing new features – poking and prodding, making sure the developers have covered all angles
… I think that roughly covers it, though each bullet is of course more fine-grained.

I would claim that the release testing would be most difficult without a computer – at least right now. Depends on how fast we make progress on the second bullet.

I would also claim that improving the automated tests would be difficult without a computer. Yes, we could compose test descriptions on a piece of paper and have someone else implement them. Semantics, perhaps, but sure – the tester part of that task is doable without a computer (the rest, then, is pure coding).

More interesting is the last bullet, the cognitive action of “testing”. The common approach, I suppose, is reactive – we read a requirement, wait for it to be implemented, try it out in a test environment and look for bugs. The alternative, then, would be to switch to a more proactive approach and catch the bugs at the design stage by analyzing requirements together with the developers, agreeing on an implementation and “pair program”, if you will, to be able to assist with unit and integration tests to ensure that the code is well-written the first time ‘round.

Consider the traditional test levels;
· Component tests,
· Component integration tests,
· System tests,
· System integration tests,
· Acceptance tests

They exist because we expect, traditionally, to find different kinds of bugs on different levels. If we switch our focus from system tests & system integration tests, where we currently work, to the design phase & preventative work … well, I’m afraid we are going to miss out on the bugs that appear when putting the system in a realistic user scenario, together with all the other bits, pieces and test data. This is more true, I believe, for the parts that have an interface to an actual human user. I wouldn’t feel comfortable, at least, to let them go without having seen (and tested) them with the eyes of a user.

This is the reason, I assume, that we have employed a session-based way of testing – to be able to analyze and critique the finished product and find the irregularities that show up when everything is connected.

I’m not sure how the “teach testing” weighs in … yeah, we could design tests from our models and requirements and have someone else operate the test environment and execute the actual tests while we monitor and mentor (without, literally, using a computer) but I guess that’s not really the point.

So … could I do an entire sprint in my team and maintain quality without a computer? Sure, but someone else would have to do parts of my job on a computer. So, in reality, no - not really."

Could you do your testing without a computer?

SBTM Case Study Talk

I recently gave a talk at a conference organized by ITQ here in Stockholm, Sweden. The topic for the conference was "Just Enough Testing", and how to apply testing strategically.


I provided the agile, exploratory, context-driven flavor with my case study of how I introduced SBTM at the system test level for a feature release of a trading application.

If you attended, or not, and want to have a glance at the slides, here they are.

A fun experience, and I think it was appreciated. The topic of exploratory and session-based testing is still unfamiliar to a lot of people, and it was refreshing to socialize, during the break after my talk, with all the attendees who "had heard of exploratory testing" and were "curious to learn more about it".

Out in the Wild

Every now and then, they let me out of the office.

Last week, I accompanied boss and colleague Michael Albrecht to a company in the outskirts of Stockholm where we conducted a course-workshop hybrid on exploratory testing in general and session-based test management in particular.

The course-part was a distilled version of AddQ's xBTM training and will be followed up by two audits during the fall.

Just the other day, I took part in aforementioned xBTM course, as part of my training to teach it. I'll be behind the podium the next time we run it - November 30.

A month before that (October 27), I'll warm up my public speaking voice at ITQ's annual conference. This year, the topic is "Just enough testing" and I'll spend 40 minutes on a case study in session-based test management.

New Assignment, Old Friends

As of this monday, I'm back in the Stockholm gaming industry. Not a huge industry, it seems, as the new office is filled with old faces. Old as in familiar.

The warm and fuzzy feeling within has grown steadily this week as I have browsed the internal wiki and come across such gems as
- there is no "tester" title within a scrum team, only a role which is responsible for driving the testing agenda
- QA is not "quality assurance" (rather "quality assistance")
- the main objective of a tester is to provide information about the product
- the tester should participate in design discussions
- all testing should be context-driven, requiring skilled testers

Fantastic!

With such an amazing framework already in place, maybe I can focus some of my free time on
1. version 1.5 of AddQ's SBTExecute - a session report parsing tool which supports the xBTM way of working and is used in our training on the subject (http://www.addq.se/utforskande-testmetodik-xbtm/)
2. progressing towards becoming an educator myself in the aforementioned training
3. prepping my talk at the annual ITQ conference on "Just Enough Testing", Oct 27

Citi's URLs

I am aware that this topic has been covered a million times already within testing circles. Apparently Citi didn't do their homework and put an account identifier in the URL after a user had logged in. Without any further authentication, anyone could simply change the identifier to access another user's account. You can read more about it here.

The NY Times spins this as sophistication on the part of the villains in this piece.

One security expert familiar with the investigation wondered how the hackers could have known to breach security by focusing on the vulnerability in the browser. “It would have been hard to prepare for this type of vulnerability,” he said. The security expert insisted on anonymity because the inquiry was at an early stage.

I love how things like this still happen. I tried to mentally defend the Citi folk by putting myself in their testers' position, thinking "hey, I could probably have overlooked a mistake like that as well - it's too simple, I'd probably begin testing something much more complex .." and so on. Also disregarding, of course, the fact that there were probably a whole group of testers involved in this, as well as audits by third parties etcetera.

The more I thought about it, though, the more I realize that no, no I wouldn't miss something like this. Even if I just spent a single test session on security testing, a flaw like this would be evident within the first five minutes. I hope so, at least, I really do - or I might as well hand in my keyboard and join the circus right now.

The other angle of this that is absolutely adorable is the alleged security experts who are interviewed by the NY Times.

The method is seemingly simple, but the fact that the thieves knew to focus on this particular vulnerability marks the Citigroup attack as especially ingenious, security experts said.

I think "ingenious" might be a strong word for this. Sure, if the number sequence isn't obviously a social security number or bank account number, it would require an ounce of imagination. An ounce. Didn't we do this to Hotmail accounts 15 years ago? Also no, it would not have been "hard to prepare for this type of vulnerability". If this isn't the first rule of handling accounts on the web, it should at least be among the top three.

Still, I love the whole thing. It bears witness of an innocence, a naïvité - as well as a fundamental conviction of refusing to learn anything from history that promises I will be able to find work as a tester for many years to come.

New Employer

On a personal note, I have recently changed employer. The choice was mine, and the reason behind it was to surround myself with like-minded testers - something my previous arrangement could not provide. Thus, since a few weeks, I am a part of the Swedish consulting company AddQ.

Short story, I love it. Suddenly, all my colleagues are experienced and committed professionals who have grazed the testing field for several years - an excellent opportunity for me to soak up all the knowledge I can handle. It also makes it easier for me to attend testing conferences and similar venues, to keep up with who's who and what's what at the forefront of testing.

I'll also try to get involved in AddQ's wide offering of testing courses and process improvement services. Closest at hand will most likely be teaching one-day classes in thread-based and session-based test management.

Another push towards further learning and valuable experiences, I hope.

Picking Raisins

I have had the good fortune to work at a customer where there is a tradition of involving support staff in testing activities. Partly because there aren't a whole lot of dedicated testers, but the benefits extend beyond pure addition of manpower.

In an organization developing a complex system, a lot of roles tend to be filled by domain experts; customers, perhaps, with long experience of using similar systems. It's easy to see how an individual with extensive knowledge of the domain would fit well into the support department or as a product owner, for instance. As a tester? Naturellement.

It might be difficult, I imagine, to find a team of people with good product knowledge and refined testing skills - or at least quite time-consuming to train such a team. So why not simple pick the finest raisins from both cakes (assuming you like raisins .. if not, reverse the analogy) and have a customer-savvy product whiz performing exploratory testing together with a tester who can add fuel to the testing fire in the form of test ideas and techniques. The tester should also take care of documentation - i.e write a session report, keep track of any deviations from the charter and file trouble reports - to maximize their co-tester's play time with the software.

I'm doing just this at my current assignment, and I love it.

Of course, allocating resources from different departments always requires some planning ahead and can be subject to sudden change - but once things fall into place, testing is a breeze. The service-minded contributors have really appreciated the level of steering that the charters provide, and the testers get an opportunity to learn lots about how the product is perceived and used by the customers.

SBTM in the Test Lab - a First Taste

Alright, wow. I've been overwhelmed by impressions lately, regarding exploratory testing and SBTM. The story behind it, briefly:

Where I work, it's about time for The Big Release. It's a new feature release of The Product, containing lots of exciting new features that the customers will adore, probably. Traditionally, each feature release has been preceded by a bug bashing session and followed by many, many bugs reported from end users.

To turn this around, the problem was attacked from two fronts; 1: to limit the scope of the release, and keep better track of any changes made to the software base and 2: to convert the uncontrolled bug bashing to a structured period of system testing.

In practice, 2 means borrowing some very talented support staff to perform the work and organizing the testing effort in a session-based manner. This has been my assignment, and it has been absolutely fascinating.

We're in the midst of if right now, and I'll return with a full review and comments from the people involved later on - but I figured I'd post a little teaser.


Some of the highlights include

• our very own test lab; a separate room with seven workstations - excellent for a bit of privacy and undisturbed sessions
• the "light-weight session-based concept" (I'll post more on the details of this later) very quickly accepted and adopted by the testers involved
• pair testing is surprisingly efficient
• using Jira as a common tool for test charters, bug reports and feature requriements is just super

The one thing that struck me the most is how quickly we got started. The whole team gathered in the test lab after a 45 minute introduction and was up and running with their first session within minutes. I had prepared test charters for the entire system test, out of which one was a "setup, configuration, familiarize with the environment" type of charter. I pointed everyone to where the charters could be found and where to take notes from the session and every keyboard was frantically clicking away before I could finish the sentence.

From past experience, selling the session-based approach is not necessarily as easy-going as this. I suspect it might be related to the fact that my current team is made up mostly of support staff, and not testers.

New Assignment - A Financial Adventure

Sometimes, some things do fall into place. I'm a week into my new assignment, trying to learn as much as I can about stock exchange, financial instruments and trading strategies - most of which is very new to me. More importantly, however, I have been hired to implement SBTM on a system test level and train the service organisation in ET. Fascinating stuff, and I'm thrilled to be here.

The customer develops and sells a stock market trading client, packed with plugins, features and market connections. They have come a long way with automated regression tests, and are beginning to realize the benefits of complementing that with manual, exploratory, testing. Initially, I will be doing that in a system test sprint early next year with the help of a team from the support department.

The support staff has been involved in testing in such a manner before, but under much looser guidance. With my help, the goal is to make the test period more productive by setting some structure, giving some coaching and establishing some form of informative reporting.

Right now, I'm in the midst of

• organizing tool support
• setting up a test lab (configuration of a number of workstation, places to sit, etc)
• putting together a test plan, writing a set of charters
• gathering input from the dev teams, assessing risks and need for regression testing
• preparing users and test environments
• establishing a process for testing, debriefing and reporting
• working on a curriculum for testing day 1, where Michael Albrecht and myself will be instructing and coaching the team in SBTM
• worrying about test environments, users, external test systems, test data, ...

Fun? Hell yes.

Metrics and Session Based Test Management

First off, props to tester superstar and my colleague Ann Flismark who just gave a talk at SAST on the implementation of session-based test management together with James Bach and Michael Albrect of AddQ. If you happened to listen to them and found it interesting, you can read more about the everyday practicalities below (the tool we use, for example).


Current focus: metrics. There's a lot of talk in the test organization on KPIs and comparisons. Having recently moved from script-based to exploratory testing, we face a problem of bringing our old metrics to our current way of working.

By "old metrics" I mean things like

• number of testcases per requirement (planned/passed/failed)
• number of automated test cases vs number of manual test cases

and so on.

Are these metrics interesting? There is no easy answer. On one hand, counting the number of executed test cases is meaningless since a test case is not really a measure of anything. On the other hand, these numbers are deeply rooted in all levels of the organization and should be treated with a certain amount of respect.

Also, it is not uncommon for my client to be involved in project with other companies and external stakeholders. They often approach the test organization with questions like "how many test cases have you planned for this feature?" or "what is the pass rate of tests for feature so-and-so?". One way would be to slap everyone around with the hard truth that we no longer count test cases, since we have none. That also means, however, that we need to educate everyone in our way of working (which, of course, is the long-term solution). Another, more instant, approach would be to provide other, comparable, metrics. Is this something we can do? Yes and no.

The fundamental requirement on us testers has been worded as "you need to provide metrics". Having pondered this request long and hard, as well as googling for "metrics in exploratory testing", "session based test management metrics", etc (and not finding much, I might add), we have arrived at a couple of conclusions.

Primo, we collect metrics that are valuable to us and that we can
1. use to better ourselves and increase our efficiency
2. show our stakeholders once they are up-to-speed on what session-based test management is all about
These include things as time spent in session, time spent setting up test environments (and other tasks), and how these evolve over time.

Secondo, we collect and compile metrics that can be translated to and compared with traditional figures. These include, for example, requirement coverage, test coverage and test session complexity.

The thought process behind producing and displaying metrics has led to a somewhat more rigid and refined test management process - something we constantly strive for.

The Process - Improved
Each sprint starts with a day of planning where we decide to gnaw our way through a number of stories. Each story exists as a high-level requirement written by a project manager and is labeled with a number.

During the test planning, I, as a tester, go through the stories and for each
1 draw a sketch of the use cases, components involved, how it ties to the rest of the system, etc, on a whiteboard
2 get the developer (and, possibly, project manager) to detail the sketch with me, point out things I may have missed, and give input on risk areas and what to focus on while testing
3 cover the sketch with post-its, each holding a charter of 1-2 sentences e.g. "test the write-to-file throttling functionality" or "regression: test the auditing of this-and-this data to backend"
4 estimate the number of sessions I will need to cover each of the charters

I then update the relevant playbook accordingly, and bring the charters into my nifty tool - a simple text file will hold my test plan:

story;charter;no of planned sessions

9;test the write-to-file throttling functionality;1

and so forth - one line per planned charter.

When I'm ready to do a session, I visit the tool and am instantly presented with the current test coverage. I pick a charter that is planned but untested (or not tested enough), and get to work (simply clicking it will give me a session report template with the basics already filled out). As I check my reports into our version control system, the test status is updated automatically.

The tool then shows me

• what stories we cover in the sprint
• what stories I have adressed by planning tests (writing charters) for them
• what charters I have planned for each story
• how many sessions I have planned for each charter
• how many sessions I have planned for each story
• what charters I have covered with sessions
• test work left for this sprint

And, of course, the good ol' stuff like

• how much time I have spent in session (and on various other tasks of my choice)
• how that time has evolved over time
• how many sessions I have spent testing a certain component or functional area

And another thing, which is something of an experiment at this stage .. session complexity. The thoughts behind it were something like "the session report contains the steps I took during the session, such as

• went through use-case UC12 with user account F18, verified audited data in table T_11K

... couldn't each such step be translated to a scripted test case? At least the headline of a scripted test case, so we can count them ...".

So now we count them. Is that good? I'm not sure. But it's comparable. If someone asks me "how many test cases have you run for feature so-and-so?" I could say "17" if I don't feel like giving the whole here's-how-exploratory-testing-works-lecture. If that number is meaningful to them, why not? I believe in taking small steps, and making sure everyone understands why something is the way it is.

So what about failed test cases? Well, we traditionally counted passed/failed test cases based on the test run during the end of our sprint. Every failed test case would result in a bug report. If that bug was serious enough it would get fixed, we would run the test case again and set it to passed. If it wasn't a showstopper, the bug report would stay open after the end of the sprint. Short-cutting that whole process, we could translate "failed test cases" to "open bug reports after end of sprint".

As always, this is in an experimental stage. We hope to be on the right path. Do you spot anything missing? Are we ignoring important metrics or measuring wrong? How do you do it?

 Just because I like visualizing, here's a bird's eye view of my SBTM tool in its entirety:

 

... with links to previous sprints on top, followed by a summary of all the time entries I have written (setup time, test time, etc). We talked about the big red-and-green graph earlier, and the little one just below shows requirement coverage (what stories have we planned for, and whether they are adressed by planned test charters). The blue graphs show trends for certain time entries (setup time and test time being the most interesting), how the time spent is distributed among tasks and by day over the sprint. At the bottom are links to all session reports, sortable by date, covered component or area.

On Improbability

This summer I have read a magnificent piece of literature entitled The Black Swan by empirical skepticist (or was it skeptical empiricist?) Nassim Nicholas Taleb. Its subtitle is "The Impact of the Highly Improbable", and this is exactly what it deals with.

I won't post an exhaustive summary or review of the book, as it has already been done quite well by others, but perhaps a quick introduction is in order for you to follow the rest of this post.

A Black Swan, in this context, is a highly improbably event. Not impossible per se, but an event for which we are completely unprepared because it lies beyond the border of our imagination. 9/11 is a recurring example, or World War 1 - or the discovery of black swans back when man knew only of white ones, for that matter.

The author is a former trader, and so a lot of the reasoning and examples come from the world of finance. The theories are however applicable to most situations - testing, for instance.

Taleb's ideas boil down to a list of concrete advice for minimizing the impact of negative black swan events; making the swans grayer. The gist of it is make yourself more aware of - or at least less intimidated by - the "unknown unknowns", the things that you don't know that you don't know. Yet. And we have all been there, yes? A sprint that doesn't quite go exactly according to plan because we didn't consider every last dependency within the system, we found a showstopper bug just a little too late, two developers were home sick for three days, priorities were changed mid-sprint for this or that reason. But we continue to plan, and we continue to fail.

In my office, I've seen a trend in moving away from detailed time estimates. We used to sit down in the beginning of every sprint, voting for hours on each task and trying to match the available hours. Now, we have an hour-long meeting every week where we go through the backlog and vote for story-points on every new story (and revise our guesses from last week). After a few sprints, we are starting to get an idea of how many points we can churn through in three weeks.

Our morning meetings have moved from crossing off numbers on post-it notes to sharing your "gut feelings" about the tasks at hand. Neighboring teams have implemented a "fist of five" or thumbs up/down voting for their sprint tasks to give the scrum master an idea of how the work is progressing.

Considering the inaccuracies of time estimates, I strongly advocate an approach where less time is spent on guesswork. We are never going to be 100% correct in our time estimates, so let's not put too much effort into them. That hurts less when we're wrong.

How do you do plans and time estimates, and what happens when the unexpected occurs?

Exploratory vs Scripted

About six months ago, the company where I consult decided to make the switch to exploratory testing. It has been an exciting journey, and I feel very fortunate to have been there along the way - learning plenty, and hopefully contributing equally.

Recently, the discussions have circled around whether the new way of working is better than the previous. A natural reaction. For instance management, as well as others who happen to read our test reports, have started to wonder about the change of information provided regarding our test results.

This review process has spawned a few highlights that I figured I'd share with you.

Streamlining the Daily Work
Is streamlining still a buzzword? Perhaps I should just call it "cutting the crap". Anyhow, we seem to agree that the actual testing activities haven't changed all that much - at least if we compare with the best and brightest parts of the scripted methodology. Allow me to explain.

With our earlier way of working - I refer to it as "scripted testing" just to give you a feel for it - the work during a sprint followed this rough chronology:

1. discuss new feature or component to be developed with project manager, tech lead and developers
2. ponder possible test cases and risk areas on a fairly high level
3. receive a non-final version of the software from the developers and
• install it in a test environment while looking for flaws in the installation procedure, associated database scripts, etc
• start the software, make sure it can communicate with other parts of the system
• use the software, see how it works in practice, take notes of possible inputs and related outputs
• distill the knowledge acquired in 3.3 into scripted test cases with clear action-result steps
4. iterate all of 3 until we have reached a version that is "ready for test" (often around when the sprint is about to end)
5. compile a test suite using the newly created test cases from 3.4 together with an assorted selection of older test cases that cover other, possibly affected, areas for regression testing purposes
6. mark test cases as passed or failed and put the results into the test report

I guess this seems somewhat familiar to most, with a few modifications here and there. Where we are now is something more along these lines:

1. discuss new feature or component to be developed with project manager, tech lead and developers
2. draw an overview of the feature or component with all paths to other parts of the system and all connections to any actors, producers, consumers, etc that have a part in the relevant use-case(s)
3. use the overview to identify risk areas, oracles, testability deficits, dependencies, etc together with developers and architects
• compile all new knowledge into a playbook for the feature or component, formulate charters to focus the test effort
• receive a non-final version of the software from the developers and
• during one or more recon sessions, explore the installability and operability of the software, find how it works in practice
• during one or more analysis sessions, following the charters defined in 4, further explore the software learning as much as possible about it - paying extra close attention to shaky/complex/unstable/risky areas that will need to be tested more carefully; also, look for possibilities to automate parts of the testing, e.g. to provide test data or parse log output
• during one or more coverage sessions, following the charters defined in 4, use all of our knowledge and skill to cover as many of the software's possible uses as possible to find as many bugs as we can
4. iterate all of 5 until we have reached a version that we (testers, project managers, other stakeholders) are satisfied with
5. compile the session reports to a complete test report for the work done during the sprint

Let's compare point 3 of our scripted methodology with point 5 of our current, exploratory, approach. These are the steps where we really put our little brain cells to use and channel all of our test expertise into finding bugs and ironing out the kinks in the software. And it is this part of our work that I claim is not all that different now. It has changed, however, and in a most crucial way. This is how:

With a scripted approach, the testing that we do is tainted by the fact that we ultimately need to compose scripted test cases, with easily re-testable action-result instructions. Naturally inquisitive as we may be, eager to explore and track down elusive bugs, we run the risk of being trapped in this mindset and restrict our testing too much.

If we enter the testing with an exploratory approach, the work will be more directed towards finding bugs rather than producing test cases. We then adapt the reporting to what we have done, rather than changing what we do to fit the reporting.

No Nonsense Reporting
We have struggled a bit with trying to get our test reports to reflect our actual work, as I have written about in the past. The old test report format, which was based on our scripted labor, had an understandable appeal in that they were easy to understand. We claimed to have executed 114 test cases, out of which 4 had failed. A good percentage, one might argue. I want to point out that, from a personal perspective, I find such measurements tremendously useless. Not only is there no record of what the test cases cover, there is also no indication as to how they have been executed. Test case instructions could have been misunderstood by the tester, or even incomplete to begin with. However, the reports were easy to understand at-a-glance, and that is one of the more important aspects for our readers, the stakeholders.

What we want to keep is the simplicity of the test report. We want the reader to be able to understand, within seconds, what the results of the tests are. At the end of a sprint, we testers usually have the best understanding of the state or quality of the software. We can tell you how complex the changes have been, how many bugs we have found, where the risks lie ... and it is that understanding that we need to distribute through the test report. Personally, I feel better doing so in other terms than in a nonsensical number of test cases.

The Loose End
Making a change, as we have, brings up a lot of questions. Particularly by those not directly involved, but who might still be paying for it in the end. Changes cost, but we do them because we hope to gain something more in the end. We have been asked things like "How is your session based testing better than what you did before?". That is hard to measure. Do we compare the number of found bugs? The number of incidents? The perceived well-being of the testers? The amount of time spent testing instead of managing test case instructions?

Pending a more thorough investigation by KPI gurus, I'm inclined to say that the last couple of things listed above are the more important. A happy tester that can spend the better part of his or her time testing will be more familiar with the software, have a better understanding for how the software can be - and is - used, and will find more bugs.

The Upside
There have been a few other positive side effects by this transition. For instance, we have started tracking our time in a more detailed way. It now takes us seconds to figure out how much of our time during the sprint that has been used for testing, or setting up environments, or reporting bugs. We could have done that without the adoption of session-based testing, but it would have been a much greater effort.

The "playboards", our whiteboard-based playbook embryo, allows us to communicate with developers, architects and testers with greater ease because we have something to talk about. We can physically stand around a common visualization and point, talk, draw and erase.

Also, we spend more time doing what we know and love - test

Big Screen Testing

Vacation is approaching, and most of Sweden has entered the traditional July coma. This could mean a slower pace and longer lunches for those of us still at the office, or it could mean using the extra time to try out some new exciting things, or to finally get done all the things that we normally can't find the time for.


Today, friday, I grabbed two testers and occupied the newly installed video conferencing room down the hall. It is equipped with two 60" screen LCD monitors and not much more. Well, a web cam and sound system, but that's not relevant. We brought two laptops and hooked them up to the screens and network and I drew a crude sketch of the system under test (an application that has received a few fixes for handling communication errors) and described the relevant scenarios to my colleagues.

We tossed up a handful of console windows for log monitoring on one screen, some tools for traffic generation on the other, and away we went!

We spent about 90 minutes in the session (and about the same amount of time setting everything up ...). I noticed about the same advantages as with the pair-wise testing that I talked about in an earlier post. It was pretty neat when we encountered a problem/oddity that we wanted to question a developer about. It awoke the interest of no fewer than three developers (did I mention the summery slow pace at work?) and they could all gather around our 2x60" screens with ease and discuss the issues. Other than that it was a rather ineffective experiment, taking into account the time it took to get everything set up ... and considering that we will probably not be able to make it a permanent installation, we will probably leave it at that - an experiment. Fun, though, and educational - I'm still very pro large screens, immersion and collaboration. A whole-hearted team effort is hard to beat.

Cinematography

Video - an excellent way of

1. recording instructions
2. explaining the outcome of a test case
3. clarifying odd behaviour in the application while testing
4. ... and more, I'm sure.

I've grown fond of CamStudio, mostly because it was among the first result when I googled for "screen recording software". Again, it's not about using the best tool, but using the one that suits you best.

Every now and then I get questions along the lines of "we saw this weird thing live where this and this happened ... can we reproduce it in a test environment?". Instead of having the inquisitor perched upon my shoulder while I try to recreate the symptoms, I will say "give me a minute or two and I'll see what I can do". Then I record the window where the weird behaviour is supposed to occur, and try to reproduce it with the bits of information I have. When (sometimes if) I succeed, I save the clip, snip out the relevant bits and send it to the person asking "is this similar to what you were expecting?". It beats a screenshot and two paragraphs of text any day.

Visualizing and Reporting Sessions

Visualization is key.

I talked about visualizing the test planning recently, with the aid of charts and graphs etcetera. That is all well and good, but not worth a lot unless we, after testing has ended, can present the results in an easily understandable way.

Something that is traditionally expected from a test report is the number of test cases: planned, executed and passed. This information is not horribly valuable, much because a "test case" as such is not a gobally recognized unit of measurement. A "test case" has no size, no value.

To say that "I have executed one test case, and it passed" says nothing of the quality of the tested product. Nothing indicates that the test case even remotely covers any of the changed code.

Additionally, an inherent problem of scripted test cases is that a single test seldom finds any new bugs. It may find a bug the first, and maybe even the second, time it's run, but by test run 98 it's a weak, regressive confident-booster at best. If you want to find bugs in the software, scripted test case will help you very little.

Our approach to visualizing test results was to illustrate the changes and risks in the software divided into logical areas and sub-components. This is then mapped to test coverage or test effort during the sprint. Allow me to examplify.

Our team develops components C1, C2 and C3 as part of a larger project. New features are planned, implemented and tested during the course of a three-week sprint.

The three components interact - with each other, and with other components in the system - and we have divided their functionality into eight logical areas; A1..A8.

Of course, the words we use are more intuitive. We call the components by name, and the areas are of type "login", "auditing", "robustness" or "registering a player". To keep is simple, I'll use the A/C abbreviations for now.

At the beginning of the sprint, we bring out the "risk" matrix:

For each change to a component, we'll make a mark in the row corresponding to the affected area. Some areas do not exist in some components.

The matrix is kept up-to-date during the sprint, to account for bug fixes or planned changes that grow unexpectedly, and gives us a light-weight "heat map" which guides us in focusing our test sessions.

In the matrix above, we have worked with areas 2, 3 and 5. We planned, for instance, two bug fixes in component 1 related to area 2.

Just like in the "visual test plan", earlier, we try to cover development activities with test activities. This heat matrix is complemented by an identical matrix where we make a mark for each test session covering a certain area in a certain component.

The result is a graph where the sum of the changes to each area is shown, and compared to the amount of testing made in the same area.

This helps us focus the testing where it matters, and we have found it to reflect our "gut feeling" of the state of testing after a finished sprint pretty well.

In addition to the test coverage information, we also display the amount of time we have spent in test sessions, reporting bugs and setting up test environments, compared to the time spent out of session. The ambition is to identify the time thiefs, and to bring the time spent in test sessions to a maximum. More on this in a later post!

Visual Test Planning

I have a habit of drawing crude sketches of any system, feature, function, component or use case that falls on me to test. These sketches help visualize the flow of information in the system, and may reveal components that affect or are affected by the actors in the use case. Primarily, I use it for myself as a "map" of the system/object/area under test after having shown it to the responsible developer and having all the question marks on it explained to me.

Now, we try to get a head start on testing by involving ourselves in the development process as early as possible, by inviting ourselves to meetings and eavesdropping on developers and architects when they discuss requirements with each other or with the project manager. During an early presentation/discussion meeting with an architect and responsible developers, and a clarification on a whiteboard, it quickly became apparent that their mental image of the, in this case, new feature came pretty close to what my crude sketch would have looked like right before I dove into testing. It seemed natural, then, to combine the two.

This map - as a concept, not in its original incarnaction - has evolved in our team during a couple of sprints, and now is a recurring presence on the team whiteboard. In lack of a better word, I'll call this map the "system overview" where the word "system" means the software which we are interested in testing right now, be it an entire application, a new feature or something else.

The system overview will then be the basis of what James Bach would call the "coverage heuristic", and the activity where we - the testers - start jotting down test targets. We identify all points in the system where information is entered, stored or manipulated. We track down all oracles - interfaces where we can access said information - via logs, in the database, through a web GUI, and so on. We flag all communication channels and ponder on how the system will and should behave if we "cut the cord" (robustness testing).

The next step will be to produce charters that cover the points describe above. Depending on the complexity and importance of any particular point, it may require more charters before we consider it sufficiently covered.

Every pink note, above, represents a charter. The orange notes are bugs that need to be investigated (black dot) or fixed (blue dot).

We then consider the number, and types, of test sessions needed to exlore the charters in a satisfying way. If the charter is new and unknown to us, it will probably require at least one recon session just to familiarize ourselves with the code. If the charter seems to be complex enough, it will probably require several analysis sessions to cover it. We also consider the likelihood of finding bugs, and account for the time needed to revisit the charter.

As with all things, practice makes perfect - after a few sprints, the ability to reliably predict the number of sessions needed will grow. It is of course important to follow up on the estimates done during sprint planning both during the sprint - to be able to warn or re-plan if the estimates don't hold - and after the sprint - to improve the estimating skills for the next sprint.

Taste Testing

The other day, I found this little tidbit which I thought fit nicely with the theme of this blog.

The note says "I have taste tested and serviced your coffee machine" and is dated and signed by technician.

Testing coffee machines or testing software, it's all the same basic concept. In this case a regression test of coffee machine functionality and test of end-user experience after having performed maintenance work on the system. Using taste buds instead of log files is just a matter of selecting the best suited tool for the job.

Guided Recon Session

We - the team - are currently working on a new application which is an extracted subset of an already existing system component. In other words, we - the testers - are already familiar with the domain, the use cases and the functionality on a conceptual level. It still needs to be tested, though.

In its earliest incarnation, I deemed the application too unstable to be the target of any real test sessions. Thus, before unleashing the band of bugthirsty testers at my team's disposal, I was fortunate enough to be able to sit with the developer for a couple of days, deploying new versions to the test environment, getting an introduction to an administrative interface, being able to ask questions and getting immediate feedback - a priceless method, if you can find the time and resources.

After these most educational sittings, and getting the application to install, start and perform according to specifications in most cases, I decided to spread my recently acquired knowledge to the other testers. Also, I feared that my close-knit relationship with the developer might have caused some of his love for his own software rub off onto me, making me more forgiving, or even oblivious,to some of the application's quirks and weirdnesses. Another two pairs of eyes would be worth their weight in gold, and more.

This led to a guided recon session with two testers performing the actual work with me in the background guiding them on a very loose leash, taking notes and answering questions.

Although not an approach I would recommend for everyday purposes - three testers on the same case is not always cost efficient - the benefits of doing this during a one-hour recon session were obvious:

•  we had six eyes on the same problem area and thus less probability of any anomaly slipping through unseen
•  it quickly became apparent that our three different views on reality and what is considered "normal" (in the domain under test, specifically) enabled one of us to pick up on bugs and other strange behavior that the others did not notice
•  the de-briefing became instant - if a tester came across something that might be worth investigating, that was immediately brought up and could be discussed (for, say, a minute) and with the added experience of the sherpa and the other tester we could quickly determine whether it was worth exploring right away, or should be recorded for further investigation during another session

This approach was also very well suited for a recon session, where problems with setup, new behavior, etc could be solved immediately by the sherpa, allowing the tester to advance more easily.

A Taste for Test

I am a tester.

Employed by a medium-sized (150-or-so employees) consulting company in Stockholm, Sweden, I currently work with software testing at a large online gaming company. The product is transaction-intensive, geographically widespread, and used by players on one end and administrators on the other, both with high demands on availability and response times.

The testing methodology we apply, and a personal favourite of mine, is largely exploratory and charter-based, drawing a lot of inspiration from The Church of Bach. At the time of writing, we have recently had a personal visit from James Bach who spent a day with us, discussing our implementation of exploratory testing, our use of charters, and delivering many helpful tips and examples. I am fortunate enough to spend my days at a very liberal workplace, where new thoughts and ideas are always welcome and the staff is well motivated and unafraid of change - if, of course, for the better.

This culture gives us testers lots of room to explore not only software, but our own processes and strategies as well. As mentioned, we are currently in the midst of chiseling forth an exploratory way of working which means running into a lot of obstacles, but also overcoming said obstacles and spawning brilliant solutions, convenient shortcuts and new ways of collaborating. In the words of James Bach, "Are you writing about this?". Well, I am now.

This blog is a window into my workday. If any of the above tickles your interest, or if you are curious about different approaches to exploratory testing or are looking for input on different kinds of test management, you will hopefully find some use for the things I write.

My intention is not to preach anything as an "absolute truth", and I don't think anyone should, but to add a trickle of thoughts to the ocean that is software testing and to, hopefully, tickle your imagination and curiosity in new ways and, primarily for those less experienced in the field, stimulate your Taste for Test.