Peformance Testing Software Services-I

I think it is important to discuss how software services are different from software programs and in particular map the differences between services and programs in the domain of software performance. So let's begin....
What is a software service? (How is it different from a software program e.g. Microsoft Excel)
Examples of software services range from google's translator to XBOX live from Microsoft, from google maps to Yahoo answers. Given that this is a diverse group, it is still not difficult to outline the properties that all of the above have in common :

• They are always ON (always there), in other words there is no down time
• They are accessible from everywhere (most respectable services are available globally, but even if you think about the US, most of these services are as available in washington state as they are in florida)
• They can support multiple users at the same time, (most of these services have thousands of users active at any given unit of time)

  To summarize , a software service is a piece of software that is always available(always in memory of some coputer server somewhere) and is accessible to all its clients (via some agreed upon protocol) and can support multiple users at the same time.

Performance of software services: Now that we know what traits software services have in common, lets ask ourselves what that means for the performance of such services.
For a service to be always ON , or always available it has to reside in memory of a computer(multiple computers) all the time, it(some instruction inside it) should always be in the ready state to be picked up by the processor, it should have the connection to all the resources (database, file shares etc) open at all times (this is not a requirement but it is an optimization that can be assumed for most services). This has some very important connotations specially with regard to physical memory. The memory used by the service should not keep increasing , in other words new memory should be used and then returned back to the OS after some time. Let me explain this a bit...if the memory usage of the service keeps going up with time it will eventually exhaust the physical memory limit for a process. (Remember memory management in OS, see windows internals by Microsoft press for a refresher). This, what i just desrcibed, is a common issue plaguing software services and it is called a Memory Leak. A memory leak is bad for any kind of software but especially services, since they are always ON ( I can't emphasize that enough)
It's not just memory leaks, imagine that we are designing a weather web service , which is accessible through a web browser. The weather service gives you the average temprature at your location by deducing your location based on your IP address (this is called reverse IP and most web portals use it). Let's say this IP-geo location map is stored in the database. Every time a user accesses the web service, the users IP address is looked up in the database and an associated location is retrieved. Since this is a service and is always ON, we would need to always have an open connection to the database (establishing data base connections is expensive).

I hope this makes you appreciate the performance traits of software services. In the next post I will talk more about the performance of software services.... so long till then.

Basics , definitions and directions

What is software performance?Software performance is a measure of three kinds of metrics
Latency/Response Time/ Round trip Time etc. This basically means how fast is your software, for example, how fast does your website's home page load on a browser connected to broadband network, or how quickly can your software service complete an operation (for example how quickly does your blog post get published once you click the publish button).


Throughput/Bandwidth: How many concurrent operations can your service (applied more to services) process at the same time. Example how many users can be active on facebook at the same time, or how many users can send an email at the same second in yahoo mail or how many people can search for some reality show star on google at the same time... (this limit is usually crossed by DOS attack).


Efficiency: How efficiently does the software use the resources available to it while working at its peak load. Resources available to a software include local machine resources ( the machine on which the software is hosted) or resources available over a network. Example how much disk space is used by youtube.com to host the latest cute kitty video. Or how much memory is used by the Browser favuorites service to serve 250 million windows live users.


Why is software performance such a big deal?
Firstly and obviously users (= money) like fast software, whether it be a web service or spread sheet on the desktop.
Secondly, and this is not one that everyone thinks about, better performance often means more optimized resourse usage, hence it lowers the cost to serve for software services.Example, facebook.com would make lot more money if they could reduce the number of servers hosting the facebook service by 5%. Remember, cost to serve is a very important to managers, and you can use this to make them see why software performance is such a big deal.


What is performance testing?
Performance testing is the process of measuring software performance, it involves the following:
1. Very Important: Having performance goals.
2. Simulating the correct conditions under which to test software, this includes loading the right amount of data, having the correct amount of load, and the correct conditions (e.g. network latencies , important for web site performance).
3. Collecting performance information (like logs, ETW traces, profiler traces etc)
4. Analyzing collected information and validating the performance against goals.


There are a lot of terms thrown around in the performance testing field and before I talk further it would make sense to try and define a few most common ones.
Definitions:
Stress Test: Testing a piece of software by incresing the load on it (example number of concurrent users) till the software starts failing.
Stability Test: A performance test that validates how long can a service run without crashing under normal load expected in the real world.
Availability: is defined as a measure of how available a software service is.
Reliability: is defined as a measure of how reliable a software service is.
The difference is a service can be available 100% of the time but might only return a valid response 50% of the time. Think of availability as a measure of 404 error in http, where as reliability a measure of 500 erroc code in http.
Resource Leak: This happens when resources used by the software are held on to even when they are no longer being used. Examples include memory leaks, thread leaks etc.




Directions:
The rest of this blog is going to talk about specific topics on performance testing, some basic and some advanced but all interesting.