The first step to starting performance testing is to have quantifiable goals for the performance of your software service. Most software teams don't fully realize the importance of having correct quantifiable and verifiable performance goals. Setting accurate performance goals is the first and the most important step to making high performance software services. Most of the times the people specifying these goals are not performance experts and I think that is where it goes haywire, I highly recommend that people with perf expertise be consulted when setting up performance goals for a software service or any piece of software.
You might be thinking, what is such a big deal about specifying performance requirements for a software service? It's easy, here it goes, the software service (let's say a online library service) should respond to all requests in under 5 seconds. There, what was so hard about that...
Let me explain.
The first thing that we missed out here is identifying user scenarios. Your software service might respond to a bunch of request types. For example lets consider a library service which is exposed over the web so that kind you can interact with it from anywhere.. A user might interact with this service in a range of scenraios. Some of the user scenarios are listed below:
- 1. Check if a book is available
- 2 Check out a book that is available
- 3 Extend the return date of a borrowed book
- 4 Add name to book's wait list
- 5 Pay dues
and so on.
Note , there be a lot more scenarios for a real service assuming that all scenarios are responded to in under 5 seconds may be ok but it would require your test team to test the performance of all scenarios which will cost time and money. Therefore we need to identify a core set of scenarios and concentrate on making sure that those scenarios are high performing, before we work on testing the other scenarios. Question is how do we identify core scenarios? Here is how: Identify the scenarios that most number of users will use at most times, in other words identify high traffic scenarios. In this case, the user scenario for checking a book's availability is much higher traffic than the scenario for paying dues. Hence we should go back to our performance requirement and make this change....rather we should qualify it by identifying a core scenario.
Change #1: The users should be able to check whether a book is available or not, in no more than 5 seconds.
Hmmm, that made sense what else are we missing.
Let's see how about the number of books in the library? Do you think its going to be faster to search for a book in a library of 100 books or in a library of 10000 books? Of course it will be get more and more time consuming as the number of books keeps on growing. ( Some geeky types migt think that a hashtable is O(1) and hence does not depend on n but you will still need to use more memory to hold the hash table and if the size is really big will have to read from disk so even for a hash table it would be mean faster response times for small number of books). It looks like we are ready to make a second change to our requirement...something like:
Change #2: The users should be able to check whether a book is available or not in a library of up to 2 million books in under 5 seconds.
Ok, so far so good, the line above looks like a pretty good requirement, are we still missing something, well to answer that we need to remember that this is a service, which means it will have more than one user, and a good chance that we will have multiple users may be requesting the core user scenario (the check book available in our case) at the same time. It is pretty obvious that a service under more traffic will be slower than a service with less traffic all other aspects being the same. It looks like we found out what we were missing, it’s the traffic to the service or as it’s called technically, throughput and is measured in requests/sec. It seems so obvious now, we should definitely qualify our performance requirement with this ... here goes....
Change #3: At 2000 requests/sec or less the service should be able to check whether a book is available or not in a library of up to 2 million books in under 5 seconds.
Phew those were quite a few changes, are we done though? Well not quite...do you think it will be ok for a service to be super fast but fail from time to time? Of course not, and that is what we are missing. Reliability.. It’s the measure of how often a service responds to a request successfully, i.e. doesn’t time out, does not send errors in response and is available to accept requests. (Availability and Reliability are actually different metrics, send failure errors count against availability, where as response errors or time out count against reliability). So let’s add reliability to our requirement..
Change #4: At 2000 requests/sec or less the service should be able to check whether a book is available or not in a library of up to 2 million books in under 5 seconds while maintaining a 99.99% reliability.
One more thing that we should keep in mind that software services that are accessible over the internet or intranet might have an occasional network hitch or some unaccounted error, therefore it is not possible to have the 5 sec response time a 100% of the times. This is where statistics comes to our rescue, we use a statistic called percentile to qualify response times. Generally we 75% or 95% values, what that means is that 75% of all the responses received are under 5 seconds. So let’s add this to our requirement...
Change #5:At 2000 requests/sec or less the service should be able to check whether a book is available or not in a library of up to 2 million books in under 5 seconds (at 95%ile) while maintaining a 99.99% reliability.
One last thing, we should also mention that the service while meeting the above requirement should utilize less than a certain %age of system resources for e.g. processor , network, memory. Most of the times this aspect is not added to the performance requirement explicitly, but i think its a great idea to do so. So the final change to our performance requirement..
Change #6: At 2000 requests/sec or less the service should be able to check whether a book is available or not in a library of up to 2 million books in under 5 seconds (at 95%ile) while maintaining a 99.99% reliability while using less than 70% of system resources.
That my friends, is what an accurate, verifiable and detailed performance requirement looks like. Next time you come across performance requirements for a feature you are working on try to see if they meet all the criteria we discussed above...
...signing off...
