SBTM Case Study Talk

I recently gave a talk at a conference organized by ITQ here in Stockholm, Sweden. The topic for the conference was "Just Enough Testing", and how to apply testing strategically.


I provided the agile, exploratory, context-driven flavor with my case study of how I introduced SBTM at the system test level for a feature release of a trading application.

If you attended, or not, and want to have a glance at the slides, here they are.

A fun experience, and I think it was appreciated. The topic of exploratory and session-based testing is still unfamiliar to a lot of people, and it was refreshing to socialize, during the break after my talk, with all the attendees who "had heard of exploratory testing" and were "curious to learn more about it".

Out in the Wild

Every now and then, they let me out of the office.

Last week, I accompanied boss and colleague Michael Albrecht to a company in the outskirts of Stockholm where we conducted a course-workshop hybrid on exploratory testing in general and session-based test management in particular.

The course-part was a distilled version of AddQ's xBTM training and will be followed up by two audits during the fall.

Just the other day, I took part in aforementioned xBTM course, as part of my training to teach it. I'll be behind the podium the next time we run it - November 30.

A month before that (October 27), I'll warm up my public speaking voice at ITQ's annual conference. This year, the topic is "Just enough testing" and I'll spend 40 minutes on a case study in session-based test management.

New Assignment, Old Friends

As of this monday, I'm back in the Stockholm gaming industry. Not a huge industry, it seems, as the new office is filled with old faces. Old as in familiar.

The warm and fuzzy feeling within has grown steadily this week as I have browsed the internal wiki and come across such gems as
- there is no "tester" title within a scrum team, only a role which is responsible for driving the testing agenda
- QA is not "quality assurance" (rather "quality assistance")
- the main objective of a tester is to provide information about the product
- the tester should participate in design discussions
- all testing should be context-driven, requiring skilled testers

Fantastic!

With such an amazing framework already in place, maybe I can focus some of my free time on
1. version 1.5 of AddQ's SBTExecute - a session report parsing tool which supports the xBTM way of working and is used in our training on the subject (http://www.addq.se/utforskande-testmetodik-xbtm/)
2. progressing towards becoming an educator myself in the aforementioned training
3. prepping my talk at the annual ITQ conference on "Just Enough Testing", Oct 27

Citi's URLs

I am aware that this topic has been covered a million times already within testing circles. Apparently Citi didn't do their homework and put an account identifier in the URL after a user had logged in. Without any further authentication, anyone could simply change the identifier to access another user's account. You can read more about it here.

The NY Times spins this as sophistication on the part of the villains in this piece.

One security expert familiar with the investigation wondered how the hackers could have known to breach security by focusing on the vulnerability in the browser. “It would have been hard to prepare for this type of vulnerability,” he said. The security expert insisted on anonymity because the inquiry was at an early stage.

I love how things like this still happen. I tried to mentally defend the Citi folk by putting myself in their testers' position, thinking "hey, I could probably have overlooked a mistake like that as well - it's too simple, I'd probably begin testing something much more complex .." and so on. Also disregarding, of course, the fact that there were probably a whole group of testers involved in this, as well as audits by third parties etcetera.

The more I thought about it, though, the more I realize that no, no I wouldn't miss something like this. Even if I just spent a single test session on security testing, a flaw like this would be evident within the first five minutes. I hope so, at least, I really do - or I might as well hand in my keyboard and join the circus right now.

The other angle of this that is absolutely adorable is the alleged security experts who are interviewed by the NY Times.

The method is seemingly simple, but the fact that the thieves knew to focus on this particular vulnerability marks the Citigroup attack as especially ingenious, security experts said.

I think "ingenious" might be a strong word for this. Sure, if the number sequence isn't obviously a social security number or bank account number, it would require an ounce of imagination. An ounce. Didn't we do this to Hotmail accounts 15 years ago? Also no, it would not have been "hard to prepare for this type of vulnerability". If this isn't the first rule of handling accounts on the web, it should at least be among the top three.

Still, I love the whole thing. It bears witness of an innocence, a naïvité - as well as a fundamental conviction of refusing to learn anything from history that promises I will be able to find work as a tester for many years to come.

New Employer

On a personal note, I have recently changed employer. The choice was mine, and the reason behind it was to surround myself with like-minded testers - something my previous arrangement could not provide. Thus, since a few weeks, I am a part of the Swedish consulting company AddQ.

Short story, I love it. Suddenly, all my colleagues are experienced and committed professionals who have grazed the testing field for several years - an excellent opportunity for me to soak up all the knowledge I can handle. It also makes it easier for me to attend testing conferences and similar venues, to keep up with who's who and what's what at the forefront of testing.

I'll also try to get involved in AddQ's wide offering of testing courses and process improvement services. Closest at hand will most likely be teaching one-day classes in thread-based and session-based test management.

Another push towards further learning and valuable experiences, I hope.

Picking Raisins

I have had the good fortune to work at a customer where there is a tradition of involving support staff in testing activities. Partly because there aren't a whole lot of dedicated testers, but the benefits extend beyond pure addition of manpower.

In an organization developing a complex system, a lot of roles tend to be filled by domain experts; customers, perhaps, with long experience of using similar systems. It's easy to see how an individual with extensive knowledge of the domain would fit well into the support department or as a product owner, for instance. As a tester? Naturellement.

It might be difficult, I imagine, to find a team of people with good product knowledge and refined testing skills - or at least quite time-consuming to train such a team. So why not simple pick the finest raisins from both cakes (assuming you like raisins .. if not, reverse the analogy) and have a customer-savvy product whiz performing exploratory testing together with a tester who can add fuel to the testing fire in the form of test ideas and techniques. The tester should also take care of documentation - i.e write a session report, keep track of any deviations from the charter and file trouble reports - to maximize their co-tester's play time with the software.

I'm doing just this at my current assignment, and I love it.

Of course, allocating resources from different departments always requires some planning ahead and can be subject to sudden change - but once things fall into place, testing is a breeze. The service-minded contributors have really appreciated the level of steering that the charters provide, and the testers get an opportunity to learn lots about how the product is perceived and used by the customers.

SBTM in the Test Lab - a First Taste

Alright, wow. I've been overwhelmed by impressions lately, regarding exploratory testing and SBTM. The story behind it, briefly:

Where I work, it's about time for The Big Release. It's a new feature release of The Product, containing lots of exciting new features that the customers will adore, probably. Traditionally, each feature release has been preceded by a bug bashing session and followed by many, many bugs reported from end users.

To turn this around, the problem was attacked from two fronts; 1: to limit the scope of the release, and keep better track of any changes made to the software base and 2: to convert the uncontrolled bug bashing to a structured period of system testing.

In practice, 2 means borrowing some very talented support staff to perform the work and organizing the testing effort in a session-based manner. This has been my assignment, and it has been absolutely fascinating.

We're in the midst of if right now, and I'll return with a full review and comments from the people involved later on - but I figured I'd post a little teaser.


Some of the highlights include

• our very own test lab; a separate room with seven workstations - excellent for a bit of privacy and undisturbed sessions
• the "light-weight session-based concept" (I'll post more on the details of this later) very quickly accepted and adopted by the testers involved
• pair testing is surprisingly efficient
• using Jira as a common tool for test charters, bug reports and feature requriements is just super

The one thing that struck me the most is how quickly we got started. The whole team gathered in the test lab after a 45 minute introduction and was up and running with their first session within minutes. I had prepared test charters for the entire system test, out of which one was a "setup, configuration, familiarize with the environment" type of charter. I pointed everyone to where the charters could be found and where to take notes from the session and every keyboard was frantically clicking away before I could finish the sentence.

From past experience, selling the session-based approach is not necessarily as easy-going as this. I suspect it might be related to the fact that my current team is made up mostly of support staff, and not testers.