When Evaluating SaaS Partners, Make Performance Assessment a Priority

This guest post today comes from Mike Horn, Senior Performance Consultant at Intechnica in the UK. Intechnica are a digital consultancy specialising in performance assurance and custom application development – preventing and solving IT performance problems, and building great web and mobile applications.

When choosing Software as a Service (SaaS) there are many criteria that we consider: availability, price, data security to name a few. But how often do we consider performance? After all, a poorly performing application has several negative knock-on effects to a business: a one second delay in web page load time decreases customer satisfaction by 16%, and more than a third of users go on to tell others about a bad experience using a website. Performance has also been shown to affect the bottom line, as Shopzilla.com showed when a five second improvement in page load times increased their conversion rate by 7-12%.

Yet a poorly performing SaaS solution could lead to considerably greater problems than an on-premise alternative. After all, we don’t own any of the infrastructure on which the service is running and we are powerless to fix the problem.  Considering the implications, shouldn’t we add Performance to our list of criteria?

But how do you assess the performance of a service? That’s where Intechnica can help and where Janrain turned when they wanted to prove to themselves and the market that their hosted registration, social profile data collection and storage service could handle any load that a potential customer might throw at it.

We started by assessing what a potential customer may require of the service and identified a number of typical user journeys.

The duration of the test, the “think time” over the user journey and the number of iterations were calculated based on Janrain’s own observations. Together they make up the “load model”.

Next we converted these user journeys into scripts that TrafficSpike, our cloud based load testing tool, could execute. The pacing of the scripts was determined by the load model.

The scripts were then launched at Janrain’s service from the cloud by a number of virtual users over a period of time to reflect a rapid increase in load such as may be experienced after a directed marketing campaign.

Once the test was complete TrafficSpike was able to show Janrain the number of realistic “hits” that Capture could handle together with metrics on data throughput and much more.

So what did we find?

  1. During three load tests, Janrain’s registration, data collection and storage service handled a transaction rate of over 800 transactions per second for login and over 600 for registration and data.  For a system that performs complex registration logic and workflow, this is an amazing metric.
  2. In a further stress test with an average response time under 0.5 seconds, Janrain was able to process 2,472 transactions per second for login and 549 for a comprehensive registration flow. 1600 virtual users were used to generate this load; 800 registering new email addresses, 800 logging on using pre-existing email addresses.
  3. Tests were performed against an encrypted data store (which introduces computational complexity).
  4. We believe these results would more than satisfy any known use case in the market today.

So, in summary, before buying a SaaS product, do the following…

  1. Know your business requirements.
  2. Make sure that the service you are buying can actually support your business and scale requirements.
  3. Agree to an SLA (service level agreement) that will protect your business in the event of a failure (No service is 100% available).