Performance testing a complex video streaming environment at speed and scale

“It was basically a best-endeavors de-risk for [the media company],” says Reuben Korngold, Consulting Director at AccessHQ. “With only six weeks, we didn’t have the time to test the way we would’ve liked, but we could reduce the risk from a high level to a more confident level. Everybody said it wasn’t possible to test HD video streaming at this scale, but the [media company’s] COO said, ‘I want to make sure that if half a million people get on and stream, they’lI be able to watch HD.’”

There were concerns that the media company’s existing load-generation tool would not be able to handle the volume, and whether other performance testing solutions would be able to provide the level of scalable cloud capacity. This was a crucial factor, as AccessHQ knew they had to push the system to its limits, simulating up to triple the number of users streaming HD video beyond what was known to be the media company’s previous peak. They could not tether themselves to any single cloud provider. Given the size and scope of the test, they needed a vendor-agnostic, multi-cloud testing solution that would allow them to run load tests across multiple data centers from a combination of Amazon AWS, Google Cloud, and Microsoft Azure.

Key obstacles included:

• Simulate HD streaming video at hyperscale (1 million+ viewers)
• Viewership expected to be 2-3X greater than anything the company had ever seen before
• High-visibility, high-reward/high-risk programming event
• Only a 6-week performance testing window, with inflexible schedule
• Highly complex system of myriad third-party integrations
• Needed vendor-agnostic, multi-cloud capability

AccessHQ partnered with Tricentis to leverage NeoLoad’s cloud platform to simulate the customer journey end to end for hundreds of thousands of concurrent users.

Says Reuben Korngold, “NeoLoad’s scalable cloud capacity was key. We used more than 1,900 load generators across AWS, Azure, and Google Cloud.” NeoLoad also natively interacts with container orchestrators such as Kubernetes, OpenShift, Microsoft AKS, Amazon EKS, and Google GKE to provision and use load testing infrastructure automatically, on demand.

“NeoLoad was the only product that we could identify with a specific video player component to drive the load testing. Some of our testers had been looking at how they could try to simulate themselves, and the fact that NeoLoad had this capability meant that we would already be getting video player metrics,” says Korngold. “The fact that we’d done another video streaming test previously in NeoLoad, albeit smaller — just some movie trailers for one of the cinema chains in Australia — also gave us confidence. We’d actually scripted and executed performance tests using the video capability before.”

“We were pushing boundaries,” Reuben explains. “We had that faith from our previous dealings with NeoLoad that these were the guys to work with who could help us make it happen. I’d just get on a call and say, ‘This is what we’re thinking — can you support us?’ If you tried that with other performance tool vendors, you’d get a level-1 help desk person. Instead, I’m picking up the phone and getting connected with the right people. We’d all get on a call and solve some very complicated issues in rapid time. We had only a six-week project schedule — it could take you that long to get a hold of the right person at another performance testing vendor. You just wouldn’t even want to try.”