How building a software testing strategy is like building a house

Almost every enterprise across the globe has made (or is making) transitions to agile development and DevOps delivery models – and for good reason. Both practices promise the creation of higher quality software delivered more rapidly. But the movement to agile/DevOps has not been a silver bullet, and sometimes has unintended consequences for teams without considering how they’ll evolve the test strategy process. In a recent Tricentis-sponsored webinar hosted by TechWell, I walked through how the test strategy document, an element of waterfall testing practices, often gets forgotten in the transition to agile testing – and make a case for bringing it back in a modern way.

A helpful metaphor for the layered test strategy approach

The traditional waterfall test strategy is often remembered as a long, tedious document that was created during an equally long and tedious meeting. However, the concept and purpose of test strategy documents remain integral to any layered testing strategy. A well-organized and clear test strategy provides everyone in your organization, from team members to executives, with a clear picture of how each part of your strategy supports and builds on the other.

With that in mind, I find that building a house is an apt metaphor for the steps involved in building an effective test strategy. You start with the blueprint (the strategy session) and end with a view of how the finished house fits into the neighborhood (the importance of end-to-end testing). If you’re interested in seeing how this plays out, you can watch the on-demand webinar, in which I break down each step of test strategy creation using this metaphor. Or read on for my responses to a few of the most insightful questions I received from the audience during the live webinar.

Answers to common software testing strategy questions

How many pages should a typical test strategy be?

If you’re in a regulated industry such as banking or healthcare, your requirements are going to be more stringent, and I advise that you include your auditors in the test strategy conversation. This way you make sure you capture the requirements they need in case there is an audit.

Otherwise, begin by asking what layers you require in your testing strategy. For every layer you have, consider if you are truly addressing the associated questions, and if you are effectively performing that type of testing. Additionally, don’t forget your goals and audience. Who is your test strategy for? What purpose is it serving?

Does test driven development (TDD) qualify as a test strategy?

It can do from the standpoint of unit testing, as TDD means defining tests before writing a single line of code. In that sense, it is a fantastic albeit difficult way to approach the overall development.

However, TDD does not necessarily help you when you put all your components together, and does not help you understand how performant your system is. So while TDD encapsulates a great way to define unit and potentially integration tests, it often falls flat for understanding acceptance automation or manual testing. You can pair it with things like behavior driven development or acceptance driven development, but it should not be the only thing in a test strategy document.

What are good metrics for understanding the effectiveness of a test strategy?

It is typically difficult to measure effectiveness from a document-writing standpoint, so you have to think outside the box and consider metrics like defect data, customer health numbers, or call center numbers to capture how things are getting delivered. Essentially, you want to capture details that help you answer, “Is our test strategy helping us understand our risk better and allowing us to deliver better quality to our customers?”

Take production defects – you can examine whether you have covered any scenarios that could have prevented those defects. You could also look at efficiency gains, how long it is taking you to get to production, and measuring differences in cycle times.

How do you combine tests without duplicating them?

You can look at chaining tests together, we see this often with performance and load testing. Instead of creating a bunch of new scripts, you could combine existing Selenium scripts and run them multiple times to create a load environment. In end-to-end testing, we’ve seen a chain of automation tests run together to create end-to-end scenarios.

In both of these instances, there is naturally going to be some level of duplication. Sometimes these are expensive and time-consuming tests, and therefore not run on every release. However, by defining your layers and understanding what the final aim is you can limit the duplication as much as possible.

About the author

Adam Satterfield, Sr Director of Test Engineering at Global Payments, has been in the software testing industry for 20 years. He has a wide background in industries such as military, SaaS, telecom, mobile, healthcare and finance. Adam enjoys leading and mentoring quality assurance teams as well as teaching testers how to find their inner testing star. He has spoken at multiple conferences and authored blog posts related to automation, DevOps and testing leadership. Currently, Adam is working within Merchant Technology at Global Payments to lead the test engineering team to drive automation and instill testing best practices within the CI/CD pipeline. He also is working on innovation within test engineering in the spaces of AI/ML and Blockchain.