Using an open source software stack means using freely available software to run your application. Typically, we’ll recommend a Debian Linux server. It’s fast, reliable, well documented, and there are many many Linux sys admins in the world that can help you upgrade, maintain and optimise it. On top of this platform, we’ll use a standard framework to implement your software. Django is a favourite. It’s a Python framework, and again, is well documented, very popular, and top quality Django programmers are easy to come by.

Years of experience has taught us that simple software works. We compose any software system – no matter how large – of small, well specified modules. Each module has one area of responsibility – your system may have a customer module, a login module and a payments module for example. Each part of the system can be worked on in isolation, thereby reducing the burden on a new programmer. They don’t have to read everything to fix or extend any area of function. Within each module, we write simple code. Simple code is easy to read, easy to fix – you don’t have to wade through pages and pages of complexity to find the block of code that speaks to the database – and is more reliable. Simplicity works. Complexity is often hiding problems, and is always a headache for the next development team, who often suggest a rewrite in favour of understanding and propagating the problem.

And finally, we write automated tests. Any new programmer coming to our system can use the tests as documentation. The tests show how the software should work, and enforce the ways it shouldn’t work. Better than that, the new programmer can run the tests, and prove that the system works. This is great news when you are extending your system to handle a new use case, and you want to know you haven’t upset any existing function. The tests are a safety net, they are a useful form of documentation, and they set a good precedent – we hope your new programmer will run them, add to them and keep them up to date.

So your shiny new software is delivered on an industry standard platform, is built of industry standard software, is written with very popular programming languages and is composed of neat, simple code. It’s protected by tests. And it is free of vendor lock-in. It’s our goal that your system can be worked on by any competent developer or development team. We know businesses grow and change. We realise there may be a time when you take all of your development in house, or you need to hire a maintenance programmer to upgrade your system. Of course, we’d really like that to be us, but we want that to be your choice, not ours.

When we decide that some piece of data, trend or pattern is worth monitoring, we start off with a shared spreadsheet which we can all quickly dump information into. For example, when we decided to start tracking what we spent our time at work actually doing, the first port of call was a Google Docs Spreadsheet. We use Google for our e-mail, so we got all of the sharing, authentication and security for free. All of the Singing Horse Studio LLP Founders then put five minutes aside at the end of each day to update the spreadsheet with a rough breakdown of what we’d done that day. Once we’d done this for a couple of months, we had data to play with. Which meant graphs (who doesn’t love a graph?), analysis, and informed decision making in our weekly strategy meeting. But this graphing, analysis and data extraction in itself became a repetitive task. Each week before our strategy meeting, we’d sort the spreadsheet, print several sheets and tabs out, and collate the new data to compare to previous weeks. Which is when we realised the data was valuable enough to us for us to spend the time writing an internal tool to do the leg work for us.

We carved out a little time from one of our developers and asked them to create a web based internal tool for our hours tracking. That developer, lets call them Johnny, created a lovely Python / Django / MySQL web app running on a VM on our internal Virtual Server. Django’s admin interface let us set up our users and enter the time records with very little time spent coding the nuts and bolts. A few days later, we had a rather lovely internal tool that allowed us to navigate our hours by project, person, date range, type of activity, and various other fields.

Heaps of information about our working hours is now instantly available on an internal web page, and we review it each week in the strategy meeting. Since all the historical data is also readily available, we also have a couple of graphs to show us how our activities are trending – do we have enough work in the pipeline? Have we met our business development targets this week? Do we have signed contracts for all the projects coming up in the next few weeks? In this way, our hours tracking has evolved from a simple spreadsheet to a really neat business tool. The way we all used the spreadsheet informed the design of the tool we wrote, and the tool continues to grow organically with our company as we find new things to monitor and track.

So why didn’t we write the tool first off and skip the spreadsheet? One of the main reasons is that we didn’t see the immense value in tracking our activities until after we had started doing it. We always try and start with the simplest solution, and writing a tool wasn’t appropriate until we knew we wanted to be able to compare our data to previous weeks. Had it turned out not to be a useful exercise, we would have been able to ditch the spreadsheet without having wasted any of our development resources. And since it was useful, the spreadsheet turned out to be a great rapid prototype, and the formulae we’d each started writing in it to give summary data turned out to be the starting point for our internal hours and projects tracking tool.

If you are starting to get to the limits of a spreadsheet based tracking system in your business, why not get in touch with us at hello@singinghorsestudio.com or + 44 1273 234690 to discuss converting this into a database driven tool that gives you the power to collate data at the click of a mouse, navigate your stored information with ease, and generate real time reports to suit your company requirements.

We all know the old adage … “Quality, Features, Time: pick two”. It’s a useful reminder that trying to squeeze too many features into too little time will inevitably impact quality. It’s also a useful reminder that focussing only on quality will lead to a reduction of the rate of feature development. And it even works to remind us that if we keep increasing the number of features, we’ll never ship anything!

The mitigations for each scenario? Work out the right number of features for the time you have available. Sprints, prioritisation and MVP help a lot here. Work out a realistic schedule for shipping something, (hint: make it as soon as MVP makes possible), and focus on the right level of quality for the deliverable you need. Heavy engineering is needed if the system you are building is planned to be a web scale app, while a faster, more lightweight approach might be appropriate for an internal tool that will only ever experience light use.

As always, delivering a successful software project is a balance of multiple factors, and the smartest thing we can do is measure and assess the impact of what we do versus what we’re planning to deliver.

So what do we do about scope creep at Singing Horse Studio LLP? Our Agile process takes care of it for us. The project management team captures the new requirement and simply adds it to the product backlog. The product backlog is the long list of features that could and / or will be included in the project. The development team is able to keep their heads down developing, safe in the knowledge that the content of the current sprint will not change. The customer gets their deliverable on time and at maximum quality.

What happens to the new feature we just added to the backlog? Nine times out of ten, the new feature is addressed in the next sprint. And since sprints at Singing Horse Studio LLP are between three and ten working days long, the wait isn’t too long. And the other one time out of ten? At the start of the new sprint, when we sit with the customer and prioritise the backlog to agree what to work on next, the customer chooses other features over the new requirement. Sprint planning is an exercise in comparing all the available features to decide what the customer would most like next. Sometimes that new feature just doesn’t seem as important when considered against all the other great feature ideas sitting in the backlog.

Working on and delivering software in short sprints means that we can respond to our customers’ changing needs and new features without the loss of control that scope creep introduces. Regularly assessing the backlog at the start of each sprint means the customer gets to choose what is most important and relevant — including all the great ideas the most recent demonstration inspired in them — but the team delivering your project can always give their best, as they have a clear and unchanging picture of what is in the current sprint.

Vertical scaling is adding oomph to the machine that an application is running on. Perhaps your e-commerce site is getting more traffic as your business grows, and it’s starting to creak at the seams. A common way to give the application a boost, is to add more RAM, more processors, more bandwidth, or more storage to your machine. Maybe you simply move your application to a new, more powerful machine.

Using vertical scaling is great for applications that can only run on a single machine. Perhaps you are using an off the shelf web shop that is fully integrated and can’t run on more than one machine at a time. Or perhaps the server you are using is underpowered for the latest version of the software you have installed.

Vertical scaling can be a quick and easy way to get your application’s level of service back up to standard. On the negative side, vertical scaling will only get you so far. Upgrading a single server beyond a certain level can become very expensive. If you already have a quad processor, moving up to eight processors can add cost quickly. And if you already have eight processors, you may start to wonder if there is an easier way. Especially if your disaster recovery strategy is to have a mirror machine at the ready at all times!

Horizontal scaling, on the other hand, is adding more servers to your application to spread the load. The simplest case of horizontal scaling may be to move your database onto a separate machine from your web server. Now your web server can simply satisfy requests, and your database server can simply crunch data.

If your application can be broken down into more modules, perhaps it has a three tier architecture – a database layer, a business logic layer and a presentation layer – then it can be spread across more servers. If your application is running on three machines and you find your database is holding you back, you can add a second machine to your database layer. Your data crunching capacity is virtually doubled. If your presentation layer is the next to slow you down, you can simply add a second machine serving web requests and split the traffic with a load balancer.

Horizontal scaling can only be applied to applications built in layers that can run on separate machines. Horizontal scaling applies really well to on-demand cloud server architectures, such as Amazon’s EC2 hosting platform. Horizontal scaling can also facilitate redundancy – having each layer running on multiple servers means that if any single machine fails, your application keeps running.

An application that has been designed from the ground up to be scaled horizontally tends to be really easy to scale down too. If you have a seasonal demand pattern, it is very easy to bring your application layers back down to run on a single server when demand is low. And you can add the servers back when demand is high again. And best of all? A horizontally scalable application tends to run on low end hardware. You tend to add low power servers into your cluster to reduce load, rather than needing powerful (and more expensive) machines to run your application.

So which scaling strategy should you use? Both are useful. There are many applications that can only scale vertically. They can only be run on a single server. You have a clear strategy choice with these applications! But, a well written application should scale horizontally very easily. An application that is designed to scale horizontally can also be scaled vertically. So you still have a choice. You can weigh up the cost of vertically scaling with an extra 4Gb of RAM vs horizontally by adding a new server in your cluster.

At Singing Horse Studio LLP, we always write our web applications with horizontal scaling in mind. It’s easy to do, it gives you the maximum choice in the future, and it tends to drive flexibility and modularity into your web application. And those things are almost never bad .

If you’d like to know more about scaling issues in general, why not listen to our podcast on the subject? Or better still; get in touch.

#1 EC2 Autoscaling

We got some demos of some very solid and practical auto scaling tools. Set some maximum and minimum system sizes, add some CloudWatch monitoring and you can easily create policies for when to have extra resources spun up or dropped out of your cluster. Very useful, very practical, and all bounded by hard limits (just in case your app becomes sentient I guess! #skynet).

#2 EC2 Spot Market

Did you know you can bid for time on unused EC2 instances? I didn’t! Set some instance criteria (small / large, high memory / high cpu, etc), nominate a price you are willing to pay per hour, and when an unused instance is available at that price, AWS will spin it up and run your app. Perfect for apps that chug their way through a queue of jobs. Of course, the instance will spin down as soon as it is required by AWS (for a customer paying full price, for example), so it’s not the right way to run your database layer. However, spot instances are perfect for batch processing that is not time critical – like a payroll run, or image stitching. Can’t wait for the right customer app to come along that could use it – it sounds like fun, and has the potential to save the right sort of app a lot of the cost of its processing.

#3 ZeeBox

Great talk by Anthony Rose, the CTO of ZeeBox. Not a lot of detail on how they use AWS, but a great app that adds ‘second screen’ functionality and interaction to your tv viewing. I really loved the concept.

All in all, the afternoon was well worth attending. We use a lot of Amazon Web Services technology here at SHS and Amazon’s armoury is constantly expanding. Today was a great way to stay aware of what is coming from the Cloud.

Hrm. If only I had listened to her sooner …

It seems that this advice can be applied to software engineering too, as one of the most important aspects of building a scalable online application (or any software architecture for that matter) is the ability to measure what’s going on. After all, if you can’t tell what your application is up to, then how on earth are you going to be able to tune it, or fix it when it goes wrong (or even notice when it’s broken!), if you’re not monitoring it?

Makes total sense, eh?

Of course it makes sense!

Which is why we recommend it to all our clients — monitor what your application is doing, otherwise you won’t be able to know when you have to put your scalability plan into place. (You do have a scalability plan, don’t you?!)

So recently we’ve been investigating different solutions for monitoring hosts and applications that we install for clients. Traditionally we’ve used plain old Ganglia for host performance characteristics e.g. CPU, RAM, disk, network i/o etc. and Nagios to warn us when certain things aren’t working how we expect e.g is that host up and running, are there 23 newsletter sign ups happening every hour, does that web page say what we expect, is that database responding to queries etc.

If something isn’t working how we expect to be working, then we get told about it. Depending on the system, the responsible engineer(s) receive an email, SMS text, or ping via their aNag app on their Android device. Sweet.

But we never really get much of an insight into how our software is performing once it goes live; we rarely get to see how our apps perform once they’re up and running.

Don’t get me wrong, we have logging throughout development/deployment/production life cycle, we run performance testing and load testing before go live, we collect stats. But nothing that’s easily harnessed and displayed via a simple dashboard.

We’ve been looking into different technologies to solve this. From one end of the scale there’s the roll your own a la Etsy and the statsd plus graphite solution, all the way to the SaaS (software as a service) solution from New Relic.

Most reporting mechanisms implement some kind of UDP fire and forget network ping which is perfect for the kind of web based applications we all love to develop; hey, it seriously doesn’t matter if our reporting metric doesn’t get through, because remember we still have logs on the actual host in case of emergency, but it’s great to see performance stats being charted on a cool dashboard in (nigh on) real time.

New Relic gives us that in-depth x-ray look into how our apps are performing — was it that pymongo query affecting throughput, or was it the network causing excessive latency? We can even drill down to the Python method/function layer in our RESTful services if that’s what it takes.

The statsd approach requires a little more forethought into what we need to do to implement measurement recording, but at the same time it gives us a higher level of flexibility in what we want to measure exactly.

In the meantime, Johnny is working on an internal Erlang product that, as part of its plugin module nature, will be able to measure, report, and take necessary action on significant events that occur within an application or that occur based on the server’s performance characteristics. I can’t wait to get this into production!

Either way, measuring performance is the ultimate goal, because without measurements, you can’t take any informed action, and without informed action, you’re effectively running your online business in the dark.

And you wouldn’t want that, in the same way you wouldn’t want a set of shelves that didn’t fit. Now where did I put that tape measure and saw …?

In case you don’t know what I’m talking about, Nose is a neat Python test discovery and runner. Essentially you can point it at your Python source code and it will automatically discover your unit tests (you do have unit tests, don’t you?!), run them, and then provide you with a report of success/failure. It’s perfect for Test Driven Development, and makes your whole development process as smooth as silk from the command line, and integrates nicely with the likes of Jenkins et al.

On top of that, Nose supports an elegant plugin model that allows the basic functionality of Nose to be extended in any direction you wish; for example there are plugins for reporting on your test coverage, plugins for profiling your Python code, plugins to integrate with Google App Engine, Django, or Komodo etc.

Thing is, there wasn’t a plugin available to specifically integrate with a project that we’re currently working on at Singing Horse Studio (more on that in a blog post to follow), but that’s ok, because we can easily extend Nose ourselves by creating our own plugin …

… and all was swimming along nicely until something rather odd happened.

Whenever I ran my tests with the auto discovery option (by simply typing nosetests), it would raise an AttributeError: ‘NoneType’ object has no attribute …

But if I chose a particular module to test (nosetests the_module_to_test) it would work fine.

Huh?!

Now the solution was trivial, but I needed Johnny’s help to notice it, because I was too caught up in the code, and couldn’t see the wood for the trees. I had named my plugin class starting with the word “Test” (it was relevant to the project we’re working on, not that I was too lazy to think of an appropriate name, honest!)

[cc lang="python"]
from nose.plugins import Plugin

class TestThingThatWeNeed(Plugin):
name = ‘test_thing_that_we_need_plugin’

def configure(self, options, conf):
“”"
Configure plugin.
“”"
Plugin.configure(self, options, conf)
self.do_that_thing_we_want = True

…
…
[/cc]

So, Nose being awesome and everything, kindly chose to auto discover my plugin class (because it started with the word Test), and thereby prevented any of the module from importing and initialising. D’Oh!

Renaming the class to something without Test in the name solved it. No more AttributeError when trying to run tests whilst using the plugin. Yay!

Now that this blog post is out there, you won’t make the same stupid mistake, will you?

Incidentally, keep a watch out for an announcement about the project that we’re writing the plugin for; if you’re into unit/continuous integration testing, you’ll love it …

Now, ultimately we want to follow a process that our clients enjoy, that allows us to finish projects within a well-defined and predictable time-scale, and that allows clients to launch products earlier rather than later to give best return on investment.

So as good Agile practitioners, we have been continually modifying and improving our process as we’ve gone along, throwing out the methods that fail (erk!), and tweaking the process where it succeeds (yay!)

Sure, there’s been times in the past where we felt a lot of improvement was deserved and required, but we now believe we’re at a point where we’re happy and confident in the way we work …

Agile Development

At Singing Horse Studio we follow an Agile development methodology. Now, Agile development means many different things to many different people, so I think I should describe the core tenets of our process, and why we believe it’s so beneficial.

High-priority work first

By always focusing on the most important task first, we’re given the greatest flexibility for reducing scope whenever the tensions increase between the other aspects of the project completion triangle — i.e. scope, resourcing, and time.

For example, a client wanting to launch a product as soon as possible, can safely cut down on the features if required, being safe in the knowledge that the most important work has been completed first. This actively supports development and launch of MVP (Minimum Viable Product) as described by Eric Ries in his lean startup techniques.

Short bursts of focused work

In the development vernacular, these are called sprints.

The goal of a sprint is to deliver one or more features or components of a project, and to do so with a maximum focus on actually delivering work.

Software development really benefits from the flow state, so we try to limit distractions and context switching during a sprint. This means that task/project planning, creative thinking about how to use the product with yellow widgets, discussions about how the product will evolve in 6 to 9 months time, modifications to the agreed sprint goal etc etc are BANNED — or at least those that involve the actual software developer!

This is why we have project managers and project teams; leave the coders to focus on getting the sprint completed before you involve them on these distractions.

Anyway, assuming all goes well, at the end of a sprint there’s traditionally a demonstration of the feature/component/sprint goal to the client and project stakeholders. It’s at this point we all go for a beer. Or two.

SCRUM artefacts

One Agile methodology that Singing Horse Studio enjoys following is SCRUM. As part of the SCRUM project management process, several artefacts are created — the product backlog, the sprint backlog, and the burndown chart.

The Product Backlog: At the start of the project, the product owner prepares a list of client requirements, and then prioritises each of these by business value. We contribute to this product backlog by estimating the cost of developing those features. The requirements are formed by features visible to end-users, as well as the technical requirements to build the product.

The prioritisation is important, because this is what allows us to focus on most important features first. Of course, once the sprint is under way, anything in the product backlog can be re-prioritised if desired.

The Sprint Backlog: When the team has committed to delivering a set of top priority features from the product backlog, those items are then broken down into individual tasks required to deliver each feature. Once tasks are planned and estimated, they form the workload required to deliver the next sprint. These tasks form the sprint backlog which makes for a simple reference for both product owner and development team to understand exactly what’s expected to be delivered at the end of the sprint.

The Burndown Chart: During the course of a sprint, the actual time taken to deliver a task can be compared with the original estimate. These actuals and estimates are recorded on the burndown chart, which shows the current progress of the development team. By closely monitoring the burndown chart, each interested party can easily visualise how the project is progressing, and can quickly rectify wayward planning or implementation.

Frequent product demonstrations

Demonstrations show the piece of high-priority work that has just been completed. This demonstration can be validated against the specification whilst at the same time soliciting feedback from the client to help assure that we’re doing the work required in the manner to which we’ve committed.

By running frequent product demonstrations, we all avoid the trap of completing a project without feedback. Imagine a year long software development project, and being told on day 365 that it’s not how the product should now work! Finding out after week one is a much better way of mitigating against a truly expensive mistake, eh?

Iterations

Marc Raibert once said “Good Writing is Bad Writing That Was Rewritten”. A similar thing can be said for code.

Kinda.

Apart from the fact that we start with good code, not bad code!

You see, we make good software the first time; we make great software by being able to iterate on it. Working on the most important things first, we evolve that core code with greater enhancements, additional features and clearing out defects as we go along. All this iteration over the codebase is made possible because we make sure our code is covered by tests …

… which leads us to …

Test-driven development

So, how can we continue iterating over the code without fear of breaking previously working code (something that QA discover with regression tests)?

By insisting upon a high level of continually tested code, that’s how. And which we ensure by adopting test-driven development.

Test-driven development is a way of writing software that makes it easily testable from the start. Hey, it doesn’t have to be done that way. In fact you can try and write code to test your assumptions afterwards, but usually you’ll find that you’ve made it impossible to test because your spaghetti code has introduced inter-dependencies, and relies too much upon expensive (in terms of execution time) method calls to live databases, networks and customer data. Ouch.

Test-driven development has the double benefit of firstly insisting on having tests that ensure everything works as expected, and secondly makes the codebase that much easier to understand and digest.

Furthermore, we never encounter the same defect twice. If we discover a problem, we first write a test to warn us of the defect, and then we write the code to ensure the test passes. It will never turn in to a defect ever again, even with non-related changes.

Continuous Integration

So, how can you be confident that your latest version of the product build actually works? Surely, just because the individual unit tests pass, it doesn’t mean your product as a whole will work successfully first time, will it?

Well, we have confidence because of continuous integration.

Any time code is checked into our repository, the product is automatically re-built and integration tests are immediately run. A green light means all is ok; a red light means something broke. And if something broke, then that becomes our highest priority thing to fix, because we always want to be in a state where we can “make the product live” and have no fear in doing so.

Code is Law

How many times have you read a wiki page, documentation, or a instruction manual about how something is supposed to work, but it never quite seems to work that way in reality?

It’s far too easy for documentation that lives outside of the codebase to fall out of sync with changes in the software. And so, because of this, we always aim to derive our documentation from the code itself. Tools like Epydoc help drive this to ensure we generate useful documentation.

Anyway, hopefully this has given you a little insight into how we work. It’s a process that has evolved; it’s a process that will continue evolving, I’m sure. Remove the fail, celebrate the win, and deliver high quality software products in the process — that’s our ultimate goal.

Human error.

It’s easy to overlook a misplaced comma in a config file, or heaven forbid coding a quick patch on a live running service. The human error is bound to creep in one day, and so you should be prepared.

And I don’t mean be prepared just for the human error. The advice I’m going to give applies just as equally to hardware or software failures. You see, it’s all about how you communicate with your customers when disaster strikes.

Whilst writing this, I’m reminded of a recent incident involving a DNS outage at MyDomain.com, an example of how not to handle a disaster. In fact, had they handled the incident better, we would probably still be hosting our DNS with them. I’ll be taking this as an example, and compare it with how Singing Horse Studio currently handle outages, some of which we’ve inherited from our time at Linden Lab, the makers of Second Life.

Don’t Panic

The main thing is don’t panic! No matter how bad it seems to be, taking a breath and stepping back from the problem is the best first step. You won’t be making it any worse by taking your time, whereas I’ve witnessed more small errors turn into bigger problems by stupid mistakes made under stress.

Assign One Person As Point

If you’re not a one-man-band then you’ll have the luxury of appointing a single person as point. It’s this person’s job to co-ordinate events, taking the lead in organising who deals with what, and facilitating communication with others.

Here at Singing Horse Studio, the person on point would be responsible for documenting events for the post-mortem (more on that later), and communicating with customers directly, but for organisations that have their own dedicated customer services department, point will simply communicate events to the customer service representatives for them to inform the customers.

The advantage of assigning someone as point means that you can allow the engineers to fix the problem without giving them any additional pressure and stress of “get it fixed now!”; they can focus on what they need to do whilst one person can handle the co-ordintation. This gives them the freedom, instead, to regularly inform management, customer services, affected departments etc on the progress of the fix. Which leads to …

Keep Your Customers Informed. Regularly.

If there’s anything more frustrating for a customer or manager, it’s the lack of information. Put yourself in the customer’s shoes:

Damn right. And being kept in the dark about progress (or lack thereof) is going to make your customer worry even more than necessary. Even communication saying “we’re still looking into it” is better than silence.

With the MyDomain DNS outage, we first became aware because of our own internal Nagios checks. A disappearing host isn’t a good sign, and it was quickly diagnosed as a DNS failure. I first checked to see if MyDomain had a maintenance window or a service status page, but unfortunately it just pointed to their knowledge base. Not a good sign.

So what’s the best way of keeping up with news as it happens? Twitter of course! And checking their Twitter stream showed that they did indeed have problems.

Now, at least they did something right by monitoring their Twitter messages and responding to them, but there was never a series of planned communication to the customers. Instead, the only communication they exposed was a reply to any question about their service being down with “I’m sorry, yes we’re working on it”. No explanation as to what the problem might be, no estimation time for a fix, no setting of expectations of future updates.

When customers are given content such as “At 7:57 UTC the main East Coast server suffered an, as yet, undiagnosed problem, causing software connectivity issues. Engineers are currently diagnosing the problem. An update will be provided within 30 minutes” they know what to expect; there’s a serious problem, it’s being looked at, and I’ll find out more in half-an-hour. Cool.

Tell your customers what’s going on, even if it’s your own fault. Saying “we messed up, but we’re fixing it” is going to give you respect, and you’ll be surprised at how easily customers will forgive you if you’re open and honest, and taking responsibility.

Learn From The Experience With A Five-Whys Analysis

As part of our Agile process, we’re always looking at ways to improve what we do, whether that’s looking at our coding practice, our project management style, or even how we make sure faults don’t re-occur. I guess that’s why we love test driven development because writing a failing test before fixing a bug should mean you’ll never have to worry about that bug showing its face in a later regression test.

One excellent method for gradually improving a process or methodology is something that we’ve learned from Eric Ries and IMVU (oddly enough, a competitor of Linden Lab), called Five-Whys which originated in Toyota’s production system.

It involves asking the question why five times to drill down to the root cause of a problem. The method then suggests you invest a small amount of time to address each of the five answers, effectively improving the process piece by piece without having to invest huge amounts of time, money or effort. It’s a very effective method, and I strongly urge you to read all the details.

Publish A Post-Mortem

And finally the most important part when you’ve suffered a disaster is to let people know. The guy on point would have recorded all the details in a timeline of events; you have your five-whys analysis and the actions you’re going to take to make sure this kind of disaster never happens again; so publish it.

If it’s an internal facing problem, then there’s obviously no need to publish outside of your own organisation, but it makes sense to distribute the post-mortem to at least those who had been effected by the disaster. If it was a public facing problem where your customers (or a percentage of) suffered, then make it a public post-mortem. Release it to the customer services department to distribute, put it on the upcoming maintenance/service status page (assuming you have one that works!), or publish it as a blog post.

Show people the results of the five-whys, even if it displays some of that human error — that’s ok, we’re all human, remember. Show people the steps you’re going to take to make sure that this never happens again — your customers will appreciate it.

But one thing, don’t put it on your Facebook wall and then expect non-logged-in users to be able to view it. Yes, MyDomain, I’m looking at your “you need to log into Facebook to see this page” post-mortem. Ho hum.

So, there you have it. What to do when disaster strikes at your scalable service. Now let’s hope you never have to put this plan into action, eh …?