First Step: Speciying the right performance goals.

The first step to starting performance testing is to have quantifiable goals for the performance of your software service. Most software teams don't fully realize the importance of having correct quantifiable and verifiable performance goals. Setting accurate performance goals is the first and the most important step to making high performance software services. Most of the times the people specifying these goals are not performance experts and I think that is where it goes haywire, I highly recommend that people with perf expertise be consulted when setting up performance goals for a software service or any piece of software.

You might be thinking, what is such a big deal about specifying performance requirements for a software service? It's easy, here it goes, the software service (let's say a online library service) should respond to all requests in under 5 seconds. There, what was so hard about that...

Let me explain.

The first thing that we missed out here is identifying user scenarios. Your software service might respond to a bunch of request types. For example lets consider a library service which is exposed over the web so that kind you can interact with it from anywhere.. A user might interact with this service in a range of scenraios. Some of the user scenarios are listed below:

• 1. Check if a book is available
• 2 Check out a book that is available
• 3 Extend the return date of a borrowed book
• 4 Add name to book's wait list
• 5 Pay dues

and so on. 

Note ,  there be a lot more scenarios for a real service assuming that all scenarios are responded to in under 5 seconds may be ok but it would require your test team to test the performance of all scenarios which will cost time and money. Therefore we need to identify a core set of scenarios and concentrate on making sure that those scenarios are high performing, before we work on testing the other scenarios. Question is how do we identify core scenarios? Here is how: Identify the scenarios that most number of users will use at most times, in other words identify high traffic scenarios. In this case, the user scenario for checking a book's availability is much higher traffic than the scenario for paying dues. Hence we should go back to our performance requirement and make this change....rather we should qualify it by identifying a core scenario.

Change #1: The users should be able to check whether a book is available or not, in no more than 5 seconds.

Hmmm, that made sense what else are we missing.

 Let's see how about the number of books in the library? Do you think its going to be faster to search for a book in a library of 100 books or in a library of 10000 books? Of course it will be get more and more time consuming as the number of books keeps on growing. ( Some geeky types migt think that a hashtable is O(1) and hence does not depend on n but you will still need to use more memory to hold the hash table and if the size is really big will have to read from disk so even for a hash table it would be mean faster response times for small number of books). It looks like we are ready to make a second change to our requirement...something like:

Change #2: The users should be able to check whether a book is available or not in a library of up to 2 million books in under 5 seconds.

Ok, so far so good, the line above looks like a pretty good requirement, are we still missing something, well to answer that we need to remember that this is a service, which means it will have more than one  user, and a good chance that we will have multiple users may be requesting the core user scenario (the check book available in our case) at the same time.  It is pretty obvious that a service under more traffic will be slower than a service with less traffic all other aspects being the same. It looks like we found out what we were missing, it’s the traffic to the service or as it’s called technically, throughput  and is measured in requests/sec. It seems so obvious now, we should definitely qualify our performance requirement with this ... here goes....

Change #3: At 2000 requests/sec or less the service should be able to check whether a book is available or not in a library of up to 2 million books in under 5 seconds.

Phew those were quite a few changes, are we done though? Well not quite...do you think it will be ok for a service to be super fast but fail from time to time? Of course not, and that is what we are missing. Reliability.. It’s the measure of how often a service responds to a request successfully, i.e. doesn’t time out, does not send errors in response and is available to accept requests. (Availability and Reliability are actually different metrics, send failure errors count against availability, where as response errors or time out count against reliability). So let’s add reliability to our requirement..

Change #4: At 2000 requests/sec or less the service should be able to check whether a book is available or not in a library of up to 2 million books in under 5 seconds while maintaining a 99.99% reliability.

One more thing that we should keep in mind that software services that are accessible over the internet or intranet might have an occasional network hitch or some unaccounted error, therefore it is not possible to have the 5 sec response time a 100% of the times. This is where statistics comes to our rescue, we use a statistic called percentile to qualify response times. Generally we 75% or 95% values, what that means is that 75% of all the responses received are under 5 seconds. So let’s add this to our requirement...

Change #5:At 2000 requests/sec or less the service should be able to check whether a book is available or not in a library of up to 2 million books in under 5 seconds (at 95%ile) while maintaining a 99.99% reliability.

One last thing, we should also mention that the service while meeting the above requirement should utilize less than a certain %age of system resources for e.g. processor , network, memory. Most of the times this aspect is not added to the performance requirement explicitly, but i think its a great idea to do so. So the final change to our performance requirement..

Change #6: At 2000 requests/sec or less the service should be able to check whether a book is available or not in a library of up to 2 million books in under 5 seconds (at 95%ile) while maintaining a 99.99% reliability while using less than 70% of system resources.

That my friends, is what an accurate, verifiable and detailed performance requirement looks like. Next time you come across performance requirements for a feature you are working on try to see if they meet all the criteria we discussed above...

...signing off...

Performance Testing Software Services-II

Now that we know the basics of software services and their basic performance considerations its time we get more familiar with the performance of software services. We will start this post by quick review of what I covered in the last post. I wrote about how software services have some traits in common (like alwyas ON etc) and how those traits ranslate to certain performance criteria that the services should meet (e.g. memory leaks, maintaining connections to resources etc). Now lets get a little further down this road....

When measuring the performance of software services we need to be cognizant of the following (this can also be treated as a glossary of terms used in this domain)
Production Workload:
This is the load on the service that is expected once the service is released to the world (production). This is generally easy to define, for example the production workload on something like google transaltor service can be a few hundred (700 , i am making a wild guess here) translation requests per second. It is important to keep in mind that the production workload for new services should be the expected traffic to the service 6 months or so into production. Since the traffic or load at the begining of the release might not be a right candidate for production workload.

Stability: The stability of the service is measured by checking for process crashes or hangs while the service is under production workload. Note, while measuring stability we are not looking for performance metrics (for e.g. if under production load a service stays in memory , does not hang or crash, for a gooad amount of time , ususally 24 hours then it is stable even if the response times are higher than expected or the resource consumption is higher than expected.). All we are interested in, when testing for stability, is whether the service functions without a hang or a crash under production workload for a long enough period (usually 24 hours).

Availability:  ist the  a measure of the the availability of the service to client requests. It does not gurantee that the service would respond to the client's request with the right response, but it does guarantee that the service will be available for a client to make requests against. For example, if a client tries to make a request to the bing maps service and can not reach the service(network timeout or web server time out ) then it should be counted against the availability of the service. However if the service returns a bad response that is not counted against its availability.

Reliability: The measure of the service to respond with the expected responses to the client requests. In the example above, incorrect responses count against the reliability of the service. Normally reliability can be measured by using a generic formula. like this , Reliability = 1- (number of response errors/total number of responses) * 100. Most services on the web have a reliability around 99.99 %.

Scalability: Since services are supposed to live forever in the wild, (read in the real world) and since the load or the number of client requests can increase at any time without an advanced notice, it is desirable that a software service be scalable. In other words, a software service should be able to handle more load by adding more hardware resources to it (if needed). For example, the number of subscribers to facebook now (2011) is around 600 million but it sure did not start this way, it started a little known service in Harward with only a few thousand subscribers. This enormous jump in the facebook service's subscribers is made possible by the scalability in the design of the service. In other words to handle the increased number of users all that needs to be done is to increase the hardware (or the number of servers hosting the web service). Scalability is one of the most important features for a service to have. Without the ability to handle large number of users your service will have a real hard time making money. And if you think that all software is scalable...you can make any software work faster by adding more resources (hardware) then you need to look up bottlenecks in software and Amdahl's law.
This brings us to the last feature on our list of software service features....
Cost to Serve: Cost to serve is a measure of the investment (money i.e. hardware costs, operations cost) that goes into supporting a service Vs The money the software service makes for you. For example, if it takes 500 servers to keep the gmail service running and takes 3 person staff working in shifts to maintain it to have the gmail service support 5million users,(to which 10 million ads are shown every week, and 1 % of those ads are clicked, and google makes 35 cents on each click) then the cost to serve
 (3 person salary + cost of servers (monthly installment))/ (Total clicks*.35). Ideally , a software service should have a low cost to serve, decrease numerator and try and increase denominator.
Note: There are no straight forward methods to bring the cost to serve down...this is a complicated measure specific to your service...but a very important one ..ask anyone who pays the bills in your place of work.

I hope the last two posts can explain the performance considerations to keep in mind when thinking about software services....
Happy New Year everyone.

Peformance Testing Software Services-I

I think it is important to discuss how software services are different from software programs and in particular map the differences between services and programs in the domain of software performance. So let's begin....
What is a software service? (How is it different from a software program e.g. Microsoft Excel)
Examples of software services range from google's translator to XBOX live from Microsoft, from google maps to Yahoo answers. Given that this is a diverse group, it is still not difficult to outline the properties that all of the above have in common :

• They are always ON (always there), in other words there is no down time
• They are accessible from everywhere (most respectable services are available globally, but even if you think about the US, most of these services are as available in washington state as they are in florida)
• They can support multiple users at the same time, (most of these services have thousands of users active at any given unit of time)

  To summarize , a software service is a piece of software that is always available(always in memory of some coputer server somewhere) and is accessible to all its clients (via some agreed upon protocol) and can support multiple users at the same time.

Performance of software services: Now that we know what traits software services have in common, lets ask ourselves what that means for the performance of such services.
For a service to be always ON , or always available it has to reside in memory of a computer(multiple computers) all the time, it(some instruction inside it) should always be in the ready state to be picked up by the processor, it should have the connection to all the resources (database, file shares etc) open at all times (this is not a requirement but it is an optimization that can be assumed for most services). This has some very important connotations specially with regard to physical memory. The memory used by the service should not keep increasing , in other words new memory should be used and then returned back to the OS after some time. Let me explain this a bit...if the memory usage of the service keeps going up with time it will eventually exhaust the physical memory limit for a process. (Remember memory management in OS, see windows internals by Microsoft press for a refresher). This, what i just desrcibed, is a common issue plaguing software services and it is called a Memory Leak. A memory leak is bad for any kind of software but especially services, since they are always ON ( I can't emphasize that enough)
It's not just memory leaks, imagine that we are designing a weather web service , which is accessible through a web browser. The weather service gives you the average temprature at your location by deducing your location based on your IP address (this is called reverse IP and most web portals use it). Let's say this IP-geo location map is stored in the database. Every time a user accesses the web service, the users IP address is looked up in the database and an associated location is retrieved. Since this is a service and is always ON, we would need to always have an open connection to the database (establishing data base connections is expensive).

I hope this makes you appreciate the performance traits of software services. In the next post I will talk more about the performance of software services.... so long till then.

Basics , definitions and directions

What is software performance?Software performance is a measure of three kinds of metrics
Latency/Response Time/ Round trip Time etc. This basically means how fast is your software, for example, how fast does your website's home page load on a browser connected to broadband network, or how quickly can your software service complete an operation (for example how quickly does your blog post get published once you click the publish button).


Throughput/Bandwidth: How many concurrent operations can your service (applied more to services) process at the same time. Example how many users can be active on facebook at the same time, or how many users can send an email at the same second in yahoo mail or how many people can search for some reality show star on google at the same time... (this limit is usually crossed by DOS attack).


Efficiency: How efficiently does the software use the resources available to it while working at its peak load. Resources available to a software include local machine resources ( the machine on which the software is hosted) or resources available over a network. Example how much disk space is used by youtube.com to host the latest cute kitty video. Or how much memory is used by the Browser favuorites service to serve 250 million windows live users.


Why is software performance such a big deal?
Firstly and obviously users (= money) like fast software, whether it be a web service or spread sheet on the desktop.
Secondly, and this is not one that everyone thinks about, better performance often means more optimized resourse usage, hence it lowers the cost to serve for software services.Example, facebook.com would make lot more money if they could reduce the number of servers hosting the facebook service by 5%. Remember, cost to serve is a very important to managers, and you can use this to make them see why software performance is such a big deal.


What is performance testing?
Performance testing is the process of measuring software performance, it involves the following:
1. Very Important: Having performance goals.
2. Simulating the correct conditions under which to test software, this includes loading the right amount of data, having the correct amount of load, and the correct conditions (e.g. network latencies , important for web site performance).
3. Collecting performance information (like logs, ETW traces, profiler traces etc)
4. Analyzing collected information and validating the performance against goals.


There are a lot of terms thrown around in the performance testing field and before I talk further it would make sense to try and define a few most common ones.
Definitions:
Stress Test: Testing a piece of software by incresing the load on it (example number of concurrent users) till the software starts failing.
Stability Test: A performance test that validates how long can a service run without crashing under normal load expected in the real world.
Availability: is defined as a measure of how available a software service is.
Reliability: is defined as a measure of how reliable a software service is.
The difference is a service can be available 100% of the time but might only return a valid response 50% of the time. Think of availability as a measure of 404 error in http, where as reliability a measure of 500 erroc code in http.
Resource Leak: This happens when resources used by the software are held on to even when they are no longer being used. Examples include memory leaks, thread leaks etc.




Directions:
The rest of this blog is going to talk about specific topics on performance testing, some basic and some advanced but all interesting.