How to determine an application’s performance SLAs for the first time

Building a new application is a very exciting process. Someone has an idea, and someone else builds off it. Pretty soon an entire team comes together to turn that vision into a reality, producing something that could impact millions of users.

There are a ton of emotions at play: exhilaration at the thought of achieving a meaningful goal, camaraderie among the people with whom you are accomplishing this task, pride at the end result.

And anxiety.

Are your users going to be happy?

A practiced Agile shop will involve customers throughout the development process to make sure that software doesn’t miss the mark from a task-oriented usability perspective. This process also works out many other fundamental issues, from architecture to user experience design to look-and-feel. It’s no guarantee that you’ll launch the product to a market of receptive users, but it definitely helps.

However, one area where it’s very difficult to determine in advance what your requirements ought to be is performance. How fast does the app need to be? Under what circumstances must those performance levels be upheld? What’s going to be acceptable from the user’s perspective?

Performance testing engineers establish SLAs (service level agreements) to make sure that apps reach certain benchmarks of performance. An SLA reads something like this:

When performing a product search, the first page of search results must be returned within 3 seconds, when measured from our test environments in New York, London, and Sydney using synthetic tests over both wi-fi and 3G networks. This value will be calculated as the average of hourly tests over a 24-hour period.

Armed with a set of performance SLAs, a performance tester has confidence that the site is meeting the needs of the audience and the business. The problem is: if you haven’t introduced the app to users yet, how do you know what they will find acceptable?

The problem with SLAs for new apps

For apps that have spent time in the market, it’s easy to determine your SLAs. You can look at user data, measure baselines, and simply observe behavior. When you start seeing users drop off — they bounce off the site, or they abandon a cart, or fail to complete a transaction — you know performance is unacceptable and you have information on which you can base your SLA standards.

However, new apps are harder. They aren’t in production. You don’t have history, nor metrics. You have to come up with an SLA with very little data.

It’s not enough to take a guess. Remember, every half-second of performance you try to squeeze out is only going to be made possible by a development effort, or an operational improvement, or an equipment upgrade — maybe all three. You can’t just decree these SLAs. You need some justification to make everyone spend the time and resources meeting your target.

So where do you start? Here are some ideas.

Method 1: Tap into your team’s experience

Chances are, this isn’t the first time that people on your team have tried to establish performance benchmarks for an application. You’ve probably got a couple of folks on the team who have been around the block once or twice.

Architects tend to have a good handle on how applications are built and operate, as well as the expectations for performance. They may be able to suggest a starting point that the entire organization agrees with off the bat, just based on their prestige and standing as a lead member of the development team. You might be able to get some good information from IT Operations experts as well, who have had experience running other applications and other sites.

You may not get the perfect answer, but whatever you get will be an educated one that will make sure your team takes performance seriously, so you can put the proper systems in place and gather more data going forward.

Method 2: Look to industry standards

For many industries, you can find benchmarks published online that describe performance metrics such as home page load times across a number of pre-determined sites.

The logic is straightforward: these other companies have presumably improved their own operations over time and have reached a level of site response times that users find acceptable. Why reinvent the wheel, when you can see exactly what users are accustomed to and base your SLAs off that?

Method 3: Execute some usability tests

Does your team run usability tests? If so, do they test performance? Usability testing is done by many organizations — especially those who practice Agile development — as a way of gathering deep, explicit user feedback. Often these tests aim to get inside users’ heads, even filming them interacting with an application and instructing them to speak out loud as they operate the application. Usability experts try to capture each thought, question, or insight vocally along the way. Other times usability testing is more under-the-covers — automatic tracking of and monitoring user behavior under different performance circumstances.

Either way, there are many ways of inserting performance testing into usability. If your usability testing is only designed to explore features and functions, ask your usability team to extend beyond that and include questions like “Did notice yourself feeling impatient or annoyed while waiting for that page to load?” This feedback can give you a great sense of what behavioral characteristics users will tolerate.

Method 4: Watch your competitors

Your competitors have probably done a lot of this exploration and experimentation already, and their data is easily accessible. Why not check it out?

It’s fairly easy to deploy synthetic users against any website and measure most transactions. Set up a few simple scripts that point to your top competitors and measure their home page load times and one or two other key experiences. It’s non-intrusive to them, and you can get some great information.

Some notes of caution:

• Don’t include any real customer data in your synthetic user scripts, for privacy purposes.
• Don’t abuse the power. Having synthetic users submitting forms with a bunch of fake data is a good way to get your IP address blocked.
• Whatever you do, don’t conduct any high-volume testing (that would basically be a DoS attack…which is illegal…so don’t do that).

Conclusion

Ultimately, when you are deciding on your SLAs you’ll have to draw a line in the sand. Make that decision and get your SLAs on the task board so that everyone knows what the expectations for performance are from the get-go. Then you can begin to evaluate how well the SLA matches actual user behavior once your site is generating traffic. By using some of the information you have available to you through the methods outlined above, you won’t have to do it blindly!

This blog was originally published in December 2015 and was refreshed in July 2021.