Manage Development Costs with Automated Regression Testing

In the early days of software development, think just 15 years ago, ratios on project teams looked something like ten developers to every tester. Why? The platforms were simpler. There was less device-specific variety. Fast forward to today. Now that ratio looks more like two developers to every tester. Big difference. With that shift comes increased cost. More resources must be devoted to ensuring software quality. Avoid it with automated regression testing.

What’s Driving Added Software Quality Costs?

In the last 15 years, devices, OS vendors and versions, app versions and server versions have all increased to match the maturity of the mobile platform. There’s no question ensuring quality in mobile products costs more now — especially if the progress of your project requires comprehensive regression tests to be run in short intervals to ensure the stability of the solution.

When weighing the often unintended impact of cost in mobile development and testing. Compare the complexity of these two scenarios:

1. Testing Mobile Architecture with a Cloud Backend

All phones are talking to the cloud. When that backend is updated, the update process on the phones is instantaneous. It’s a much simpler, less expensive process to manage what’s deployed in the cloud.

Verdict: Testing is manageable. Additional measures to mitigate the time/effort testers spend aren’t needed.

2. Testing Native Mobile

In this case, backward and forward compatibility become much more important. There might be two versions of the server and four versions of the app in use. Before a release, you need to guarantee that every combination works. Regression tests must be repeated eight times over and again for every subsequent release.

Regression drivers: New OS ships, new server ships, new app ships

Verdict: You’re buried in the economics of testing. Suddenly, the focus isn’t on the software — it’s hiring more bodies to keep up with it effectively doubling and tripling your project costs to keep release times short.

There’s less of an issue managing quality costs to scale when testing mobile architecture. Manually testing a highly variable solution often throughout the development and delivery life cycle is where the rubber really hits the road.

How Important Is Software Testing, Really?

Some might say, “Hire better developers, and cut out extensive testing altogether.” Good luck. Independent minds are needed in software development to approach the product with an eye on “breaking it.” Without that, the value of the product decreases.

The tester/developer relationship is much like that of the editor/writer. Writers shouldn’t proof their own work. They WILL miss things. They’re too close. The same can be said of developers. The accountability between testers and developers is a peer-level relationship. Both minds are needed to deliver success.

The Complexities of Testing Mobile

Historically testing mobile hasn’t been just different. It’s been harder. Why?

Carriers. Back in the early days of mobile, the same phone could be sold to different carriers. Then each added a different firmware version, causing the app to behave differently depending on the network.

Unpredictable GPS behavior. Testers drove in and out of coverage testing apps — racking up miles and frustration.

A few other reasons testing mobile offers more challenges…

1. Traditionally a more manual process

2. Limited tools to do automation for mobile

3. Dozens of operating systems and languages to test in (BREW, J2ME, BlackBerry, Pocket PC, Palm OS, Symbian, Windows Phone, iOS, Android, C++, Java, Objective C, C#/.NET, Swift)

The intricacies of mobile testing continue to push the strength of software teams — from inconsistencies across Android devices to technical issues in specific hardware/versions of the UX. The best software teams use grey box testing.

Sniffing in Software Development

Many of the most burdensome defects are hidden beneath the surface in the backend database. SQL queries can help crack the code by analyzing the database to learn what’s inside and explore how the data is being stored.

Greybox testing allows you to:

• Uncover defects
• Save on testing time
• Make developers’ lives easier

We recommend the Charles and Fiddler web debugging proxy applications to explore traffic between a machine and the web, HTTP caching, compression and security for potential issues.

Read: 6 Software Testing Best Practices

Scale Up, Stretch the Timeline or Explore Automated Regression Testing?

We know testing mobile is harder. Teams using a manual testing strategy have two options when the project scope grows:

• Keep high regression coverage and increase the time for testing
• Spend the same amount of time on testing and reduce the regression coverage

For teams who answer a resounding, “Neither,” mobile test automation offers budgetary and personnel relief. With automated regression testing there is no need to reduce testing coverage to expedite a release timeline. Automated tests are fast and can be run frequently — cost-effective for software products with a long maintenance life. New test cases can be added to the existing automation in parallel. Automation allows developers and software testers to work in parallel. As developers are building the solution, software testers are building the automation to test it.

Does Piling People on the Problem Work?

Manually repeating these tests is costly and time-consuming. Once created, automated regression testing can be run over and over again at no additional cost. Beyond that, they are much faster and more accurate than manual tests, when configured properly.

Here are a few of the cloud solutions available to test software solutions:

• Xamarin Test Cloud
• AWS Device Farm
• Sauce Labs
• Testroid
• SOASTA

Testing in the cloud offers benefits in test quality and cost. Automated regression testing can be run on the full range of devices. Testers are no longer limited by the size of the device cabinet or budget to upgrade. The devices are equally accessible to all members on your team — distributed or internal. Read our eBook.

Image courtesy of robuart.