Currently, a lean view on testing documentation prevails in software development projects. As the Agile methodology has taken the software development world by storm, an increasing number of businesses are adopting an Agile approach to testing documentation. The question arises how testing documentation fits into Agile and how it differs from its role in more traditional, waterfall projects. The answer is not black and white.

We look to explain that for you today. For the sake of this article, we have conducted comprehensive research on the topic and have come up with illuminating insights. If you’ve gotten curious, scroll down and read for your pleasure.

In one of the previous blog posts, we have already discussed a recipe for healthy Agile documentation. So if you haven’t seen it as yet, get down to reading it. By and large, this article dwells upon the bare bones of Agile documentation. This time, however, we are going to give a detailed account of software testing and the role of Quality Assessment (QA) in Agile. Additionally, the article will compare Waterfall vs. Agile methodologies from the perspective of testing documents types to be used in each of them.

A bit of history

Nearly 70 years ago in 1947, the first computer bug was found in a Mark II Aiken Relay Computer at Harvard University. In fact, it was a real insect: a moth trapped between the relays. The term ‘bug’ had been used before, but after Grace Hopper wrote in her diary “first actual case of bug being found” the term became really popular, and we are still using it today. If you have a few minutes to spare, you can check the article about the very first recorded computer bug in order to get more details on this funny origin.

Sadly, no record has been made of the moth’s identity for its considerable contribution to the world of technology, although it has achieved a certain immortality of its own thanks to a yellowing strip of Scotch tape.Graham Cluley

Independent computer security expert, public speaker, blogger,

Staggering examples of top software bugs

Perhaps it goes without saying that software testing does matter a lot in the modern world. I would like to back up this statement with vivid examples. Let the facts speak for themselves.

In 1996 an unmanned Ariane 5 rocket launched by the European Space Agency exploded just forty seconds after its lift-off from Kourou, French Guiana. The Ariane 5 had cost nearly $8 billion to develop and was carrying a $500 million satellite payload when it exploded. The problem consisted in a simple bug in the rocket control systems. They wanted to save on software testing, but it cost them a pretty penny.

Another bug broke Youtube’s video view counter at one point. When YouTube was first launched, nobody thought that a video would ever be watched two billion times, or rather, more than 2,147,483,647 times – the maximum numerical value that can be stored by a 32-bit signed integer. The popularity of the“Gangnam Style” video by Psy a South Korean pop star had broken a record. Google fixed this YouTube bug by changing the view count to a 64-bit signed integer. It was not a serious bug. No one got killed. Nothing exploded. It was rather an inconvenience. Here is what YouTube said in a Google+ post about it.

PayPal error has made a man an accidental quadrillionaire. In 2013 Chris Reynolds received an email regarding his PayPal account and he was stunned to see a 17-digit balance there. When Reynolds decided to double check his account later on, though, he noticed that PayPal had already fixed this bug. CNN reports that PayPal said in a statement, “This is obviously an error and we appreciate that Mr. Reynolds understood this was the case.” “I’m a very responsible guy,” Chris Reynolds told. “I would pay the national debt down first. Then I would buy the Phillies, if I could get a great price.”

These are some of the world’s famous computer bugs that cost or could have cost (if immediate action was not taken) millions of dollars. It kinda sucks, doesn’t it?

The role of Quality Assessment in Agile

Traditionally in many software development teams, QA and development work independently. At least it used to be so. They rarely interact with each other and don’t even sit in the same part of the room. It isn’t the best approach. In order to be indeed agile and provide a higher quality product in a timely manner, testers and developers should be in sync, working in parallel.

Project Managers want it to work. Developers try to make it work. Customers hope it works. Testers know how it works.StickyMinds

Testers should work alongside the development teams, so they can fully understand the requirements and functionality of the software. This would allow them to be always aware of the changes to code, requirements and product decisions. Besides, getting continuous feedback from testers accelerates the whole process and reduces the risk of delivering a low-quality product. Testers look for not only bugs but also for missing features. They have a strong sense of quality all around. The main difference is that in Agile, testing starts simultaneously with the development phase. Each feature is tested as it’s developed, not at the end of the sprint, and certainly not at the end of the project. During the same sprint, developers can make changes to a feature and give those changes directly back to a tester team for further testing. In waterfall projects, however, testing starts after completing the development phase.

It occurs to us that it is necessary to specify the role of testers in Agile. Here are several important thingssoftware testers should do when working with an Agile team:

  1. Attend sprint-planning meetings.
  2. Be present at a daily stand-up.
  3. Test throughout a sprint.
  4. Meet with developers for short demonstrations.
  5. Attend sprint retrospective.
  6. Deliver continuous documentation.

Agile testing is closely connected with exploratory testing that focuses on personal freedom and responsibility of the individual tester.

Test cases are not created in advance as in scripted testing where you design test cases first and later proceed with test execution. On the contrary, in exploratory testing, testers are involved in minimal planning and maximal test execution. Exploratory testing is also known as ad hoc testing. But in no way should you associate it with sloppy or careless work. It is a testing approach that allows you to apply your ability and skill as a tester in a powerful way and does not impose rigid limits on you. Thus, exploratory testing is simultaneous learning, test design, and test execution. If you’ve gotten interested in the topic, please watch a short video of James Bach, a prolific software tester, author, trainer and consultant, speaking on exploratory testing in the Agile world.

Testing documentation in the Agile manner

In fact, documentation makes many testers wince.

In Waterfall approach a lot of test documentation is produced which needs to be maintained.CLICK TO TWEET

At the beginning of a project, testers have less time for testing. And by the end, they have a hundred things to pick up and the entire focus is on testing.

There is another opinion concerning this issue:

As the only ones who know what they know and do what they do, many SMEs see formally documenting their test cases as a complete waste of time.Mike Hodge

Content Specialist, Lighthouse Technologies

So, some testers consider documentation to be a waste of time. They consider themselves to be testers, for crying out loud! It is the job of technical writers to document things. Other feel rather depressed when it comes to documenting test cases because it makes them slow and cumbersome. What can be done about that?

The Agile methodology gives a solution and turns the process of writing testing documentation into a lightweight journey. In Agile, you work in short sprints and focus to deliver small features by the end of each sprint cycle taking the MVP (Minimum Viable Product) approach.Agile focuses on continuous testing. Thus, you end up having less documentation.CLICK TO TWEET

To provide a common set of standardised documents, the IEEE (The Institute of Electrical and Electronics Engineers) developed the 829 Standard for Software Test Documentation.

According to this, in traditional waterfall projects, there are three phases of software testing in which the following eight document types are used:

1. Preparation Of Tests

  • Test Plan: Plan how the testing will proceed.
  • Test Design Specification: Decide what needs to be tested.
  • Test Case Specification: Create the tests to be run.
  • Test Procedure: Describe how the tests are run.
  • Test Item Transmittal Report: Specify the items released for testing.

2. Running The Tests

  • Test Log: Record the details of tests in time order.
  • Test Incident Report: Record details of events that need to be investigated.

3. Completion of Testing

  • Test Summary Report: Summarise and evaluate tests.

In traditional testing, a lot of time is wasted documenting detailed tests that are neither needed nor useful. Agile proposes a lean approach towards testing documentation. It means that heavy documentation can be reduced by using smart techniques like mind-mappingonelinerschecklists and matrices.

Test plans do exist in Agile, but these are not long documents written in isolation by a single tester, but interactive discussions captured on a whiteboard or in a mind map, so that everyone in the team understands what is going to be tested and by which method.

Test cases are also used, but they are lightweight with the minimal amount of information needed for a knowledgeable tester to execute the test. These cases are a simple description of the test without detailed steps or expected results. This makes them very quick to write.

On the other hand, instead of writing heavy test cases, you can also opt for preparing a checklist which lists all critical checks you need to do. Applying checklists and one-liners instead of writing fifty-step test scenarios reduces wasteful documentation. A one-liner is a test case that describes what needs to be performed in the test in a single line of text.

Checklists provide a cognitive net. They catch mental flaws inherent in all of us – flaws of memory and attention and thoroughness.Atul Gawande

Author of The Checklist Manifesto

In order to measure the progress of a project, test matrices are very often used in software testing. A test matrix is a kind of spreadsheet that suggests tests and captures test results by laying them out in the form of a table. In other words, it is a measurement of the work we have done on the project to improve our process and to check where we are lagging. There are test process and test product matrices. A test process matrix provides information about preparation for testing, test execution and test progress. The test product matrix gives information about the test state of the product so we can evaluate the product’s test state and indicative level of quality.

Final words

With the technical field expanding so quickly, it’s vital to keep one’s finger on the pulse of the industry and understand what is relevant at any point in time. I hope I have managed to shed some light on this topic and help you find an answer to the question raised at the beginning. Have you ever used any kind of lean testing documentation? I invite you to share your experience in the comments section below.