The Secrets to Realizing Regression Testing Best Practices

Why tie up a highly-qualified software tester in rote tasks? Let a computer do it. Automation, one of the top regression testing best practices, is one way teams can efficiently perform quality assurance while keeping project costs low.

Unlike mere mortals, automation can quickly tell you whether new features had unintended consequences on other aspects of your code by implementing regression test scripts. Automating regression tests eliminates the need for testers to manually comb through and interact with the code to verify changes didn’t create unintended consequences elsewhere in the operation. Free your testing team to think bigger, planning the automation and setting up the environment where tests can be executed.

Automate to great

Software testing didn’t make sense in the old world. It was a process performed once at the end of development. There was no reason to spend the time and effort setting up the automation scripts for a one-and-done process.

Spending the time on automation when running Agile software projects makes a ton of sense. The ROI changes dramatically if regression testing is required by your client every night, or on a regular basis. Ideally, highly-trafficked paths through the solution (or “happy paths”) should be tested each time new features are added to check for unexpected reactions or breakage.

The benefits of automation

Benefit #1: Increased test coverage, limited time spent

Often it takes herculean effort to sufficiently test the coverage of software projects. Read: Frequent repetition of the same or similar test cases performed manually. Hello monotony, goodbye efficiency. Some examples include:

• Regression testing after bug fixes or further development
• Testing of software on different platforms or with different configurations
• Data-driven testing (running same test cases using many different inputs)

Benefit #2: Better quality software

Automating your regression tests is one of the biggest wins to ensure continuous system stability and functionality while changes to software are made. Automated tests perform the same steps each time they are executed. They never forget to record detailed results. What this means for your team? Shorter development cycles coupled with better software quality.

Benefit #3: Optimize speed & efficiency while decreasing costs

Automation allows teams to keep project costs low and test coverage high by reducing the number of people teams need to test. Instead of rote tasks, your test team is focused on high-value strategy and building the automation scripts.

Benefit #4: Catch bugs earlier in the process

The only thing worse than not making your release date is finding a huge landmine in your software a few days from launch. The best way to avoid this series of unfortunate events? Perform regression testing as frequently as possible, so there are no unpleasant surprises. Automated tests give teams the ability to run more regression tests and catch bugs earlier on in the process.

Benefit #5: Improve team focus

Automated tests give teams the opportunity to focus on new implementations rather executing repeatable actions. Testers become less concerned about whether or not they receive and validate thousands of sent emails. Instead, they can devote their mental power to understanding users and improving their overall experience.

Regression testing best practices

Best practice #1: Use an element ‘id’ to locate items on screen.

The original automation tests were based on recording actions and playing them back. They might simulate the effect of clicking on a mouse at a certain pixel’s location. If the button is moved (one of many UI changes that result from the highly iterative and exploratory practice of Agile methodology), the automated test would no longer click in the correct location.

In this way it was possible to create highly fragile test code that was costly to recreate when seemingly “minor” UI changes were made. It just wasn’t feasible to re-record the tests. The cost to update the tests began to exceed to cost of the change. Pretty soon UI decisions were being made to avoid breaking the automation tests. Big. Problem. The tail was wagging the dog. Validation tests can’t begin to dictate the experiential success of the solution.

Was it true? Had testers become the enemy of momentum and exploration in software development? Every change was impactful. None were trivial. There had to be a better way. Testers needed a way to develop tests like the product itself was being developed. They needed to abstract references to components in the design and on the screen.

Remember that button called out in the automation test using pixel location? Calling it by name would allow the tests to continue regardless of where it moved in design. This shift happened much sooner in the web world driven by HTML standards. Everything on the screen had a name. It was a more difficult changeover with native apps.

Best practice #2: Have a clear understanding of your solution’s user personas.

In setting up mobile automation, user personas are critically important. After identifying the core groups of users, the features and interactions in their happy paths through the solution should be automated. That way, your team can spend their mental aerobics on the edge cases — often 10x the number of happy paths. (Happy paths are the most trafficked paths and interactions by core users through your app.)

Best practice #3: Hire team members who have embraced the shift to mobile automation testing.

Mobile testing represents a shift in expertise many teams just aren’t ready for. Writing the automation scripts involves a new skillset. Mobile automation also represents a philosophical shift for teams. It must be implemented at the very beginning of a project. The initial costs to build the automation may seem difficult to justify for teams just beginning to use automation.

Here’s how it pays off. We explore the risk and return:

The risk (investment)

• Increased cost to build and perform first test
• Increased cost to maintain working tests

The return

• Cost to repeat existing tests on old app versions against old server versions drops dramatically
• Cost to repeat tests throughout development cycle for new features plummets
• Predictability for release cycles improves
• Continuous integration now possible

Breaking down an investment in automation

Assume it takes 10 minutes to perform manual regression tests at a cost of $11. Costs scale linearly as more tests are needed.

20 features * 3 devices * 2 OS * 1 Server * 1 App = $1,320
25 features * 3 devices * 3 OS * 2 Server * 2 App = $9,900
30 features * 3 devices * 4 OS * 3 Server * 3 App = $35,640

Especially on projects where teams must test and account for a high degree of variability in devices, operating systems, servers and app versions — mobile automation is a very efficient bandaid.

Let’s say developing automated test cases for the code base after refactoring has been completed will take 156 hours. A sample breakdown of the investment might look like the allocations on the next page. Though, note, the total hours depends on the size and complexity of the solution you are testing.

13% Framework — Establish automation framework that will be the foundation of the test

77% Regression test cases — These are the most critical test cases that must be performed for each build delivered to QA

5% Test cases — Add automation test cases into continuous integration

5% Execution reports — Create the execution reports for your test cases