Vertical scaling is adding oomph to the machine that an application is running on. Perhaps your e-commerce site is getting more traffic as your business grows, and it’s starting to creak at the seams. A common way to give the application a boost, is to add more RAM, more processors, more bandwidth, or more storage to your machine. Maybe you simply move your application to a new, more powerful machine.

Using vertical scaling is great for applications that can only run on a single machine. Perhaps you are using an off the shelf web shop that is fully integrated and can’t run on more than one machine at a time. Or perhaps the server you are using is underpowered for the latest version of the software you have installed.

Vertical scaling can be a quick and easy way to get your application’s level of service back up to standard. On the negative side, vertical scaling will only get you so far. Upgrading a single server beyond a certain level can become very expensive. If you already have a quad processor, moving up to eight processors can add cost quickly. And if you already have eight processors, you may start to wonder if there is an easier way. Especially if your disaster recovery strategy is to have a mirror machine at the ready at all times!

Horizontal scaling, on the other hand, is adding more servers to your application to spread the load. The simplest case of horizontal scaling may be to move your database onto a separate machine from your web server. Now your web server can simply satisfy requests, and your database server can simply crunch data.

If your application can be broken down into more modules, perhaps it has a three tier architecture – a database layer, a business logic layer and a presentation layer – then it can be spread across more servers. If your application is running on three machines and you find your database is holding you back, you can add a second machine to your database layer. Your data crunching capacity is virtually doubled. If your presentation layer is the next to slow you down, you can simply add a second machine serving web requests and split the traffic with a load balancer.

Horizontal scaling can only be applied to applications built in layers that can run on separate machines. Horizontal scaling applies really well to on-demand cloud server architectures, such as Amazon’s EC2 hosting platform. Horizontal scaling can also facilitate redundancy – having each layer running on multiple servers means that if any single machine fails, your application keeps running.

An application that has been designed from the ground up to be scaled horizontally tends to be really easy to scale down too. If you have a seasonal demand pattern, it is very easy to bring your application layers back down to run on a single server when demand is low. And you can add the servers back when demand is high again. And best of all? A horizontally scalable application tends to run on low end hardware. You tend to add low power servers into your cluster to reduce load, rather than needing powerful (and more expensive) machines to run your application.

So which scaling strategy should you use? Both are useful. There are many applications that can only scale vertically. They can only be run on a single server. You have a clear strategy choice with these applications! But, a well written application should scale horizontally very easily. An application that is designed to scale horizontally can also be scaled vertically. So you still have a choice. You can weigh up the cost of vertically scaling with an extra 4Gb of RAM vs horizontally by adding a new server in your cluster.

At Singing Horse Studio LLP, we always write our web applications with horizontal scaling in mind. It’s easy to do, it gives you the maximum choice in the future, and it tends to drive flexibility and modularity into your web application. And those things are almost never bad .

If you’d like to know more about scaling issues in general, why not listen to our podcast on the subject? Or better still; get in touch.

#1 EC2 Autoscaling

We got some demos of some very solid and practical auto scaling tools. Set some maximum and minimum system sizes, add some CloudWatch monitoring and you can easily create policies for when to have extra resources spun up or dropped out of your cluster. Very useful, very practical, and all bounded by hard limits (just in case your app becomes sentient I guess! #skynet).

#2 EC2 Spot Market

Did you know you can bid for time on unused EC2 instances? I didn’t! Set some instance criteria (small / large, high memory / high cpu, etc), nominate a price you are willing to pay per hour, and when an unused instance is available at that price, AWS will spin it up and run your app. Perfect for apps that chug their way through a queue of jobs. Of course, the instance will spin down as soon as it is required by AWS (for a customer paying full price, for example), so it’s not the right way to run your database layer. However, spot instances are perfect for batch processing that is not time critical – like a payroll run, or image stitching. Can’t wait for the right customer app to come along that could use it – it sounds like fun, and has the potential to save the right sort of app a lot of the cost of its processing.

#3 ZeeBox

Great talk by Anthony Rose, the CTO of ZeeBox. Not a lot of detail on how they use AWS, but a great app that adds ‘second screen’ functionality and interaction to your tv viewing. I really loved the concept.

All in all, the afternoon was well worth attending. We use a lot of Amazon Web Services technology here at SHS and Amazon’s armoury is constantly expanding. Today was a great way to stay aware of what is coming from the Cloud.

Hrm. If only I had listened to her sooner …

It seems that this advice can be applied to software engineering too, as one of the most important aspects of building a scalable online application (or any software architecture for that matter) is the ability to measure what’s going on. After all, if you can’t tell what your application is up to, then how on earth are you going to be able to tune it, or fix it when it goes wrong (or even notice when it’s broken!), if you’re not monitoring it?

Makes total sense, eh?

Of course it makes sense!

Which is why we recommend it to all our clients — monitor what your application is doing, otherwise you won’t be able to know when you have to put your scalability plan into place. (You do have a scalability plan, don’t you?!)

So recently we’ve been investigating different solutions for monitoring hosts and applications that we install for clients. Traditionally we’ve used plain old Ganglia for host performance characteristics e.g. CPU, RAM, disk, network i/o etc. and Nagios to warn us when certain things aren’t working how we expect e.g is that host up and running, are there 23 newsletter sign ups happening every hour, does that web page say what we expect, is that database responding to queries etc.

If something isn’t working how we expect to be working, then we get told about it. Depending on the system, the responsible engineer(s) receive an email, SMS text, or ping via their aNag app on their Android device. Sweet.

But we never really get much of an insight into how our software is performing once it goes live; we rarely get to see how our apps perform once they’re up and running.

Don’t get me wrong, we have logging throughout development/deployment/production life cycle, we run performance testing and load testing before go live, we collect stats. But nothing that’s easily harnessed and displayed via a simple dashboard.

We’ve been looking into different technologies to solve this. From one end of the scale there’s the roll your own a la Etsy and the statsd plus graphite solution, all the way to the SaaS (software as a service) solution from New Relic.

Most reporting mechanisms implement some kind of UDP fire and forget network ping which is perfect for the kind of web based applications we all love to develop; hey, it seriously doesn’t matter if our reporting metric doesn’t get through, because remember we still have logs on the actual host in case of emergency, but it’s great to see performance stats being charted on a cool dashboard in (nigh on) real time.

New Relic gives us that in-depth x-ray look into how our apps are performing — was it that pymongo query affecting throughput, or was it the network causing excessive latency? We can even drill down to the Python method/function layer in our RESTful services if that’s what it takes.

The statsd approach requires a little more forethought into what we need to do to implement measurement recording, but at the same time it gives us a higher level of flexibility in what we want to measure exactly.

In the meantime, Johnny is working on an internal Erlang product that, as part of its plugin module nature, will be able to measure, report, and take necessary action on significant events that occur within an application or that occur based on the server’s performance characteristics. I can’t wait to get this into production!

Either way, measuring performance is the ultimate goal, because without measurements, you can’t take any informed action, and without informed action, you’re effectively running your online business in the dark.

And you wouldn’t want that, in the same way you wouldn’t want a set of shelves that didn’t fit. Now where did I put that tape measure and saw …?

Human error.

It’s easy to overlook a misplaced comma in a config file, or heaven forbid coding a quick patch on a live running service. The human error is bound to creep in one day, and so you should be prepared.

And I don’t mean be prepared just for the human error. The advice I’m going to give applies just as equally to hardware or software failures. You see, it’s all about how you communicate with your customers when disaster strikes.

Whilst writing this, I’m reminded of a recent incident involving a DNS outage at MyDomain.com, an example of how not to handle a disaster. In fact, had they handled the incident better, we would probably still be hosting our DNS with them. I’ll be taking this as an example, and compare it with how Singing Horse Studio currently handle outages, some of which we’ve inherited from our time at Linden Lab, the makers of Second Life.

Don’t Panic

The main thing is don’t panic! No matter how bad it seems to be, taking a breath and stepping back from the problem is the best first step. You won’t be making it any worse by taking your time, whereas I’ve witnessed more small errors turn into bigger problems by stupid mistakes made under stress.

Assign One Person As Point

If you’re not a one-man-band then you’ll have the luxury of appointing a single person as point. It’s this person’s job to co-ordinate events, taking the lead in organising who deals with what, and facilitating communication with others.

Here at Singing Horse Studio, the person on point would be responsible for documenting events for the post-mortem (more on that later), and communicating with customers directly, but for organisations that have their own dedicated customer services department, point will simply communicate events to the customer service representatives for them to inform the customers.

The advantage of assigning someone as point means that you can allow the engineers to fix the problem without giving them any additional pressure and stress of “get it fixed now!”; they can focus on what they need to do whilst one person can handle the co-ordintation. This gives them the freedom, instead, to regularly inform management, customer services, affected departments etc on the progress of the fix. Which leads to …

Keep Your Customers Informed. Regularly.

If there’s anything more frustrating for a customer or manager, it’s the lack of information. Put yourself in the customer’s shoes:

Damn right. And being kept in the dark about progress (or lack thereof) is going to make your customer worry even more than necessary. Even communication saying “we’re still looking into it” is better than silence.

With the MyDomain DNS outage, we first became aware because of our own internal Nagios checks. A disappearing host isn’t a good sign, and it was quickly diagnosed as a DNS failure. I first checked to see if MyDomain had a maintenance window or a service status page, but unfortunately it just pointed to their knowledge base. Not a good sign.

So what’s the best way of keeping up with news as it happens? Twitter of course! And checking their Twitter stream showed that they did indeed have problems.

Now, at least they did something right by monitoring their Twitter messages and responding to them, but there was never a series of planned communication to the customers. Instead, the only communication they exposed was a reply to any question about their service being down with “I’m sorry, yes we’re working on it”. No explanation as to what the problem might be, no estimation time for a fix, no setting of expectations of future updates.

When customers are given content such as “At 7:57 UTC the main East Coast server suffered an, as yet, undiagnosed problem, causing software connectivity issues. Engineers are currently diagnosing the problem. An update will be provided within 30 minutes” they know what to expect; there’s a serious problem, it’s being looked at, and I’ll find out more in half-an-hour. Cool.

Tell your customers what’s going on, even if it’s your own fault. Saying “we messed up, but we’re fixing it” is going to give you respect, and you’ll be surprised at how easily customers will forgive you if you’re open and honest, and taking responsibility.

Learn From The Experience With A Five-Whys Analysis

As part of our Agile process, we’re always looking at ways to improve what we do, whether that’s looking at our coding practice, our project management style, or even how we make sure faults don’t re-occur. I guess that’s why we love test driven development because writing a failing test before fixing a bug should mean you’ll never have to worry about that bug showing its face in a later regression test.

One excellent method for gradually improving a process or methodology is something that we’ve learned from Eric Ries and IMVU (oddly enough, a competitor of Linden Lab), called Five-Whys which originated in Toyota’s production system.

It involves asking the question why five times to drill down to the root cause of a problem. The method then suggests you invest a small amount of time to address each of the five answers, effectively improving the process piece by piece without having to invest huge amounts of time, money or effort. It’s a very effective method, and I strongly urge you to read all the details.

Publish A Post-Mortem

And finally the most important part when you’ve suffered a disaster is to let people know. The guy on point would have recorded all the details in a timeline of events; you have your five-whys analysis and the actions you’re going to take to make sure this kind of disaster never happens again; so publish it.

If it’s an internal facing problem, then there’s obviously no need to publish outside of your own organisation, but it makes sense to distribute the post-mortem to at least those who had been effected by the disaster. If it was a public facing problem where your customers (or a percentage of) suffered, then make it a public post-mortem. Release it to the customer services department to distribute, put it on the upcoming maintenance/service status page (assuming you have one that works!), or publish it as a blog post.

Show people the results of the five-whys, even if it displays some of that human error — that’s ok, we’re all human, remember. Show people the steps you’re going to take to make sure that this never happens again — your customers will appreciate it.

But one thing, don’t put it on your Facebook wall and then expect non-logged-in users to be able to view it. Yes, MyDomain, I’m looking at your “you need to log into Facebook to see this page” post-mortem. Ho hum.

So, there you have it. What to do when disaster strikes at your scalable service. Now let’s hope you never have to put this plan into action, eh …?

#1 Multitiered Application Architecture

Writing your application in multiple tiers allows for horizontal scaling. For example, a three tier application will typically have a backend data store layer, a business logic / application layer, and a front end layer serving the data out to your web page, application, or mobile app – or all three! These tiers should be independent of each other, so that when the database tier starts creaking, you can add a couple of hosts to that tier, replicate your database horizontally onto those hosts, and triple the capacity of the data layer. If it was the application layer, add the boxes there and run a few more instances of the application. If your application is evenly utilised, add a box to each tier to increase your capacity.

The gold standard here is to achieve linear horizontal scaling. Add one new hardware box to any of your application layers, and increase the capacity of that layer by one unit. This approach makes sure you can scale from a two box cluster, right up to a three thousand box server farm. Factor in the increasing power of hardware over time for a given cost, and you can achieve greater than linear scaling.

#2 A Capacity Plan

Your beautifully architected app won’t scale at all unless you have hardware (or cloud capacity) to scale it onto. Ideally, you’re ready for increasing capacity. You know where the budget’s coming from, you know where the capacity will be introduced, you know your cloud host can bring it online, or your data-centre can rack and cable it within two days. When you have your capacity plan, you’re ready to satisfy the increased needs of your application. Feature #4 will help you predict this need, and feature #3 will help you get it up to speed once it’s online.

#3 Architected for Easy Deployment

The easier it is to bring up an element in your system, the quicker and cheaper it is to add those new boxes. If, for instance, you’re running Ubuntu and you’ve packaged your application up into a private apt-get repository, then installation is as easy as “sudo apt-get install my-web-app” or “sudo apt-get install my-database-schema”. The installation process will initiate and launch the service, and you have a one step install, that any competent Linux user can perform.

#4 Monitoring

How do you know when your application needs to scale up or down? Because analysis of your historical monitoring data and identification of the trends it shows tells you! Basic monitoring tells you how loaded each piece of hardware in your infrastructure is. So you can see that your database host is holding your application back. Better monitoring shows you how each tier in your application is performing. So you can see that your application tier is only using 10% of the capacity of the blades it’s running on, while your front end web servers are maxed out. Migrate some of those blades to your web severs, and you’ve just increased the capacity of your application without buying any new hardware.

Conclusion

There are a tonne of other factors to writing a scalable web application. For example, graceful degradation, knowing when to make an API idempotent, knowing when to vertically scale, knowing whether to shard your database, or use a master / slave pattern, and a lot of other things. However, if you’ve got the four points above covered, you’re off to a great start.

We also have further information in our podcast – ‘How To Scale Your Online Application‘.

In scalable systems development it’s all too easy to focus on the highs. The plaudits are there for people who can claim ‘My system can scale to 200 thousand bazillion customers! Simultaneously!’. Yet not all businesses function like Facebook, Amazon or Twitter. Regular businesses suffer peaks and troughs of activity.

Limbo

Efficient handling of low activity should be of equal import to that of high activity, otherwise the overheads imposed by your ability to handle peak are going to kill your efficiency during non-peak times. The lean times are exactly when being efficient is the most meaningful. Sure, nobody likes to think about shrinkage and lean times, but that’s the reality of business – you take the lows with the highs. Taking those lows is a lot easier when your systems work with you, rather than against.

Everybody loves parfait^H^H^H^H^H^H examples

So your business is ticking along nicely. You release a product now and then and they’re as successful as you predict. Go you! Out of nowhere you have a monster success on your hands, totally out of proportion with what you expect. Party time, right?

Well, that depends if you can you clear the High Jump. If your systems were organically grown with a specific workload in mind then that unanticipated spike could cause your systems to react badly. You’re busy patching up your systems, apologising to your customers for slowness and errors, basically working your arms off when you should be kicking back celebrating. ‘Victim of your own success’ suddenly has new meaning to you.

So let’s assume that your business has a kick ass scalable system and can totally cope with that spike. Go you! But wait. What happens when that spike tails off? If your systems don’t efficiently scale back in line with the workload then you’re costing yourself money. The leaner the lean times, the more those costs are going to hurt. ‘Victim of your own success’ now has a subtle new twist.

For your systems to be able to Limbo when the time comes, they need a good sense of self. I don’t mean Terminator style machine based violence, not that you can fault their sense of ruthless efficiency, but your system needs to know when it’s time to scale things back and save some overhead.

Scalability isn’t all about scaling up, it’s about efficiently handling the workload, high or low. Now you can party

In a later post I’ll go into some of the ‘hows’ and ‘whens’.