A real story of system testing
I've done system testing for like 10 times. It's been 5 years since I joined the current team, our team release twice a year, and before every release, we do system testing. Well, actually I do. The bad part about my system testing is that we have no requirement, so no body knows what to do in system testing. This is also the good part, no body is able to tell whether I have done a good job or not.
Do not get me wrong. I work hard. All right?! Every time before I do system testing, I google for it, I read about it, I go through all testing results from the past. But there are too many guideline, rules, practices, checklists in these system testing papers, none of them includes a real story. Take a look at system testing in wikipedia, Secion "Types of testing to include in system testing", this section includes almost all of the testing techniques that I have heard. How am I supposed to finish all these? I usually get 2-3 weeks, with 2-3 developers for temporary help.
Another resource I look into is my system testing results from the past. These results show that how my understanding of system testing evolves. The first 2 years, I was a big fan of test cases. I grouped all test cases from all components together, combined them into bigger cases, every case must involve at least 2 components, then I executed one by one. This was my definition of end-to-end testing. The problem about this method was that it failed to reveal new problems. All bugs that I found were the bugs that escaped from component regression testing.
Later 2 years, I became a fan of var checklists and scenario testing. While planning, I divided different roles in our system, including normal user, manager, admin, etc, then found their interests and their tasks. With that, I designed user scenarios. A scenario would look like:
- manager
- interest : manage users
- tasks: assign user a task, assign user some resource , check on his/her status
Using var checklists, e.g. Big Var List, me and 2 developers defined complex steps and data. This did help us to find a lot of bugs, even some security bugs using fuzzy character. The problem about this method was that it was still function tests, all the bugs we found could be found in component regression tests.
I just finished my system testing 2010. This time I use a different method - risk analysis. While reading "pragmatic software testing" by Rex Black, I become a big fan of risk analysis. In chapter 4, he introduces nearly 20 kind of risks - ways that a product can fail. I draw a mind mapping of all these risks, and take it everywhere I test. But mapping all these risks in our system would generate too many test ideas, so me and 2 developers decide to set several filters.
The first filter is based on change, we assume all new features include more risk than old features. Within all new features, we set second filter, based on interaction, we assume the more components a feature interacts, the more risks it includes. Within that feature list, we set our third filter, massive impact, we assume the more user a feature impact, the more risks it includes.
With that list, we label 3 priorities:
priority C: all new features
priority B: all new features that interact with more than 1 components or environment or platform
priority A: all priority B features that might impact massive users
For priority A features, we consider not only functions, but througput - many users access this feature concurrently, volume - many user logs/data generated, load - long time high usage ,etc.
For priority B features, we consider failure mode - access component A from B while A is stopped, error handling - shutdown database while commit, and recovery - system reboot when disk full, because we know normal reboot is covered somewhere else.
For priority C, well, we didn't actually get to priority C because priority A&B cost all of our resource in the past 3 weeks. But we consider it as a good thing, we do find new ways to find new problems, we do testing from different perspectives, and most importantly, we do spend most of our resource on the highest risks.
For example, one of our feature might increase our online user accounts, double the current number or tripple it. So we re-configure our performance test, raise the concurrent session, re-run the performance case, and our system crash, no new session are accepted. I open a severity 2 bug for it. This is just one of the interesting bug that we find.
I got to say I like this method. But still, there is 1 problem. While reviewing our testing results, we find that there are several components are not covered at all, because of the priorties. So next time, I might combine all these 3 methods together, just to make sure that everything is covered, even covered once is better, right?!
