Social Services Agency
Democratizing test automation for quality @ speed at a government social services agency
Company overview
Government organizations around the globe are consistently looking to deliver innovative, often life-altering functionality to their citizens faster while accommodating stringent quality and compliance requirements. This was certainly the case for a large government agency responsible for making billions of dollars in payments each year to its country’s citizens.
Each update to these critical systems requires intensive scrutiny since mistakes could cost millions and would land the agency in the news for all the wrong reasons. The IT and development teams must ensure 20+ systems are updated and integrated properly after each release to ensure case managers and citizens alike can complete transactions rapidly and reliably. However, their legacy cadence of quarterly releases frequently required 8 weeks of manual regression testing. This delayed releases and throttled the pace of innovation. Moreover, when occasional emergency releases were required, it was extremely difficult to complete the expected level of testing in the condensed timeframe.
A product owner within the organization knew automation was the answer to their testing bottleneck. They set out to create an ROI for test automation to sell the business on its value. A known obstacle would be upskilling existing domain experts to create and run automated tests. As they noted, “You cannot just suddenly tell someone ‘Here’s C#, start programming.’”
- Industry: Government
- Products:
Challenges
- Automation was perceived as cumbersome and costly, so business cases were required for each automation effort.
- The existing testing team focused on business domain expertise, not on test automation.
- Core regression tests could take upward of 3,000 hours when run manually.
- End-to-end scenarios commonly crossed 20+ applications and technologies.
- Defects in production could result in headline news and missed payments to citizens.
Solution
In 2011, the current test service manager set out to remove the testing bottleneck, and he knew that an easy-to-use test automation solution would be key to achieving that goal. Tricentis Tosca was an early frontrunner in their tool evaluations. After a highly successful proof of concept and extended pilot, Tosca became the tool of choice to lead an internal centralized testing initiative, starting with regression test automation.
Manual testers quickly learned Tricentis’ model-based test automation approach and begin automating the large suite of regression tests with a minimal learning curve. This government agency was able to release faster and more frequently, progressing from quarterly release cycles to 6-weeks and eventually down to 2-week releases today. The solution paid for itself in less than two years. “The number of times we run the regression suite is vastly increased when we are releasing so much more frequently,” remarked the product owner of automation engineering and environment management.
Providing an additional speed boost, distributed test execution (i.e., Tricentis DEX) allowed the regression suite to be run across 35 machines, cutting test execution time in half. Moreover, Tricentis’ open architecture enabled them to streamline synthetic test data generation via REST APIs, saving 5-10 minutes on each test’s run time.
Quality has also seen significant improvements over the past few years. “We used to regularly find A, B, and C priority defects,” explained the technical test automation manager.” Today, we might go 10 releases in a row without ever finding one significant defect. In general, our quality has improved a lot over the last eight years.”
In addition to testing faster, this government agency has also increased testing transparency. A small team of their developers created a self-service portal that made testing more approachable to all within the organization and created a push to “automate more” across all levels of testing. This involved building a custom UI called the “Automation Portal” to allow anyone, from business stakeholders to QA staff, to quickly trigger test cases, which are then executed through the Tricentis distributed execution (DEX) server.
Today, this government agency’s mature testing organization continues to expand their test automation success and optimize their approach to understand where they can “go faster, test sooner, and reduce risk to the business.”
