The golden rule with software test automation is to do no harm. Using automation in your arsenal of software testing techniques can become unwieldy fast with teams opting to promote test scripts for each new feature into the suite. But, for businesses who commit to managing the effort and want to build a testing suite that offers more robust reporting, automation is the best option to scale the complexity of testing efforts without drastically increasing your staff.
Follow these 4 best practices to create a manageable automated testing suite that can evolve with your organization.
1. Don’t Underinvest in Your People
Many industry veterans have no doubt experienced the following scenario. An organization decides to build a testing client to execute a suite of test scripts. Applause for the commitment to automation software testing techniques. Now equal the commitment by hiring the right people to maintain it. In a past life, I worked with a company who committed to building automation. But, that was the extent of the investment.
They staffed the initiative with interns and a variety of other junior staffers with the know-how to run the test script but not debug them. This left the business without the confidence they hoped the testing suite could provide. The staffers couldn’t determine whether the failures were coming from the product or a flaw with the test script itself.
2. Carefully Design Your Automation Testing Suite, and Commit to Maintaining It
Managing your suite is an art form. It’s easy to justify adding scripts but unbearably hard to agree on removing them. Why? Removing scripts increases risk. Testers are rarely, if ever, scolded for writing too many test cases. It’s easier instead to recommend adding scripts when bugs arise to ensure “they never happen again.”
This can grow to be a double-edged sword, lethal if not dulled. In my career, I’ve encountered automation testing suites bloated by thousands — all selected into our “regression suite.” In this particular scenario, management of the automation effort had passed through multiple hands until the current leader and team members had little understanding of the chaos lurking under the hood.
Instead, propose a maintenance strategy as a part of the automation suite launch. Because we needed to mitigate inheriting an overwhelming volume, ours included listing the different test categories (device platform testing, device port testing, new server feature testing, server stabilization and deployment testing among others) and assigning a leader to each functional area.
Stabilization Testing Example
- Begin with automation testing to source functional issues in the user interface and validation of new mobile applications.
- Supplement automation with manual testing for features that are not exposed through the UI, new features and deeper testing around interactions with new features.
- Identify defects from the automation “first pass,” and don’t start the second pass until defect fixing is complete.
- Target 5-10 days for each pass.
Regardless of the strategy and criteria throughout, the goal is to use the criteria as a test base to measure against in determining how to best reach goals.
By properly architecting and maintaining your test suite, you can guard against creating a testing suite so large and convoluted amending it becomes more work than testing the software manually.
The worst outcome of automation is the effort to amend or adjust test cases begins dictating the evolution of your solution.
The goal of test automation is to enable efficiencies in change, not stymie them.
3. Architect Your Test Suite to Be Agile
The best-engineered test suits are data-driven. On the spectrum of automation tasks, setting up playback scripts rates as the “easiest” to create, portable scripts rate in the center, and data-driven scripts rate as the most complex.
Avoid the temptation to “record and playback” (manna to time-strapped teams) and take the time to engineer the suites correctly from the get-go. Data-driven suites use common core routines to accomplish simple tasks. Rather than relying on recordings or internal functions, they are adaptable by configuring a database or spreadsheet.
Data-Driven Scripts Should:
- Port seamlessly across machines, environments, operating systems
- Allow one engineer to keep up with the change volume of 10 developers
4. Automate the Most Strategic Part of Your Product
For many teams, this means areas under long-term active development or with multiple releases planned. That way we would see long-term ROI from our automated software testing techniques.