Exploratory Testing

Imagine that you’re a tester preparing a huge amount of scripted test cases covering everything in a system. Now imagine executing every one of them and finding out that they all pass. You open a bottle of champagne to celebrate, and then suddenly, a bug appears from some edge case scenario that wasn’t covered by any of the scripted tests. Exploratory Testing helps us go outside the box to prevent such situations and improve the test plan and overall system knowledge.

A Closer Look at Exploratory Testing

Exploratory testing is a software testing approach that emphasizes the freedom and responsibility of the individual tester. Exploratory testers optimize the value of their work by treating test-related learning, test design, and execution as mutually supportive activities that run in parallel throughout the project. This approach to testing is specific yet flexible, and its purpose is to investigate potential opportunities that might otherwise not be discovered.

Exploratory Testing vs. Scripted Testing

Compared to scripted testing, exploratory testing allows testers to take different paths during their testing sessions. It significantly improves the chance of catching bugs that could have gone unnoticed during manual scripted testing.

Ideal Exploratory Testing Scenarios

Exploratory testing is best suited for the following scenarios:

• When the project requirements and specifications are incomplete
• When there is limited time
• When verifying that previous testing has found the most critical defects

Exploratory testing fits well in Agile development practices. This testing type is suitable for testing sessions that focus on new features or areas of major change. The findings from the exploratory session can be reviewed and discussed in a meeting, and the team may benefit by creating new tests, which can be automated and/or added to the regression suite. The important thing to note is that the value of exploratory testing lies in the bugs it finds and its ability to improve the user experience and generate ideas for additional areas of testing.

Exploratory Testing Components

Charter

An exploratory testing charter describes and tracks the purpose of the testing session. The testers fill the charters.

Exploratory Testing charters include:

• Exploratory Testing Charter Purpose: A summary of what the session is designed to accomplish.
• Testers: The testers cooperating within the session.
• Date: Date of the testing execution
• Timebox or Duration: The amount of time that the testers have for the test execution
• Target or Scope: What will or will not be tested during the testing session.
• References or Files: This is an optional field used to provide the testers with additional information that may be helpful during the testing session
• Tester Tactics or Tester Charter:  Different approaches that can be used during the testing session. If a single feature is going to be tested by multiple testers, it is a good idea for each tester to have a different approach and mindset.

Using Exploratory Testing in a Project

Team-Based Exploratory Testing

It’s often a good idea to include the whole development team in the exploratory testing process so that the entire project can benefit from it. These are some examples of how the team can benefit from this type of testing:

• Easy to learn — Exploratory testing can help teams better understand the product and ask the right questions in the early testing stages.
• Multiple perspectives and viewpoints — Different perspectives can bring up different opinions, leading to a variety of useful questions.
• Quick to start up and scale — Creating a test plan with the proper test suites and cases is efficient but time-consuming to prepare. When time is of the essence, an entire team can prepare for exploratory testing by having test charters assigned to each member.

User Acceptance Testing

User acceptance testing is often carried out by people who aren’t familiar with test case syntax and are focused on testing specific features or functionalities. Teams can benefit from exploratory testing by allowing user acceptance testers to perform the business flows they know well as well as create a charter that summarizes their findings. This helps user acceptance testers focus on learning how to document proper defects without going through each step of the test case.

User acceptance testers are often not used to testing full time and have a limited window for testing and providing feedback. Including exploratory testing in user acceptance testing has the potential to significantly optimize the available time for those testers, who, with the charter, can provide feedback for possible bugs or problems they might have encountered.

Pair-Based Exploratory Testing

Brainstorming between paired testers usually leads to more constructive ideas. Also, there is a lesser chance of others interrupting during paired-based testing. Let’s discuss the two most popular pair-based testing formats that can be applied to exploratory testing sessions:

• Operator & Observer — The operator executes tests, and the observer observes the actions, the behavior, the most problematic areas, and the areas with the most usability issues. The observer writes down notes which can be discussed and reviewed afterward.
• Operator & Navigator — In this format, one person operates while the other gives instructions on which path to take and which areas to explore. The navigator is also responsible for documenting the tester’s actions and the system’s behavior during these actions.

Time Box

A Time Box is a testing session during which the tester should not be interrupted by any other activities. The ideal duration of a Time box for an exploratory testing session is between 60 to 90 minutes. It can, however, be expanded or shortened, depending on the project and/or situation. The acceptable range is 0.5 to 2.5 hours. This session encourages testers to react to the response from the system and prepare for the correct outcome.

Exploratory Testing Drawbacks

• Exploratory testing is not a systematic approach and it is difficult to determine what part of the system is covered. Teams need to strike a balance between exploratory testing and scripted testing. The biggest advantage of exploratory testing is the freedom and creativity it provides; it is also its biggest disadvantage.
• If applied in risky situations, exploratory testing requires domain knowledge of the tested system. For example, in cases where exploratory testing is being considered prior to a risky release, it should be done by people already familiar with the system. Novices may slow down the process due to a lack of understanding of the application’s expected behavior. They may also miss out on testing key parts of the application.
• The documentation process can be tedious. During an exploratory testing session, testers are responsible for documenting each step they take and its result. Compared to scripted testing, this is a disadvantage. With scripted testing, the entire test plan is laid out in advance in the form of test cases. Documentation, then, is simply a matter of reporting the results of each of these cases. This drawback is often handled by using recording tools to ease the documentation burden.

Conclusion 

On its own, exploratory testing has significant drawbacks, yet the flexibility, scalability, and utility it provides when combined with scripted testing make it an excellent fit for Agile development practices. It helps teams identify potential problems and bugs on time and provides them with a better understanding of the product they are developing and testing.