The Blog of Ian Mercer.

Continuous Integration -> Continuous Deployment

There was some interesting discussion today around the topic of continuous deployment (pushing incremental builds to production rapidly rather than batching them up.)  Here's some thoughts on that topic:

Personally I think you should be able to make a fix and deploy it to production in a matter of minutes but that you should do this rarely and it depends on the type of fix.

During the early stage of a lean startup you can go without a staging environment: the minimum is a continuous integration server deploying to a local development environment and then the ability to manually push to production as needed. (Note: I consider continuous integration a bare minimum.)

How do you define quality?  The formula I have for perceived quality is:-

Sum (Severity of bug x number of times experienced by users)

Clearly data-loss bugs rank high on severity and should be avoided before you get to production but for many other classes of bug IF you can fix it quickly the overall user impact can be minimized more effectively by having a faster build-deploy cycle than by having a perfect test suite.  And since the perfect test suite is in any case impossible, you may as well invest in improving your build-deploy cycle process first.  In an ideal world no user ever experiences the same bug twice and no two users experience the same bug.

This means however that you need a rock-solid build and deployment system that you trust as much as you trust the compiler.  (And, btw if you have this, the need to pull configuration settings into XML files that you can edit on production servers goes away: if you trust your build and deployment system you can make the change in code and rely on the process to make the change in production.  As an added benefit you now also have an audit trail as to who changed what and when and you can lock down your production environment so that very few people have direct access beyond deploying new bits to it using the prescribed process.  As we all know most problems in production are caused by human error and if you allow people to make random changes there it's hard to figure out who/what/when and where for any critical error).

Another corollary of this approach is that you need in depth logging and exception reporting.  You need to be able to understand what caused a bug to appear and how to reproduce it from a single instance of it happening.  Your logging around an exception should include the entire state (which file, which user, cookies, referrer, steps leading up to it, ...).  You should record exceptions in a database so you can see which ones are most common and can correlate their occurrence to any changes you made to the site. Your error reporting also needs to encompass the javascript running on your customers computers with every javascript exception reported back to your site using a web service.  After all, my formula is 'bugs experienced' NOT 'bugs experienced that we happen to know about'!

Another trick a lean startup can use for deployment is to employ Subversion as a binary version control system.  i.e. your continuous integration server does a build and then checks the binaries into a different Subversion tree.  To deploy (a no-database changes fix) you simply do an SVN update on the production server.  It's fast, efficient and most importantly atomic (unlike XCOPY).  It also provides an immediate roll-back capability - simply go back a revision.  Another advantage is that you can apply fixes to just one file (e.g. an image or html file) by updating just that file and can be sure that no other files changed in the process.  And, again it gives you a complete audit trail so you can see how any file has changed over time and relate that to any changes in exceptions being logged.

So, in summary: major ui changes, database changes should be pushed to production infrequently and in a very controlled fashion but minor ui changes, critical bug fixes, ... can happen all day long all the time if you have the right process in place.

Related Stories

My love/hate relationship with Stackoverflow

Stackoverflow is a terrific source of information but can also be infuriating.

Ian Mercer
Ian Mercer

Xamarin Forms Application For Home Automation

Building a Xamarin Forms application to control my home automation system

Ian Mercer
Ian Mercer

Websites should stop using passwords for login!

A slightly radical idea to eliminate passwords from many of the websites you use just occasionally

Ian Mercer
Ian Mercer

VariableWithHistory - making persistence invisible, making history visible

A novel approach to adding history to variables in a programming language

Ian Mercer
Ian Mercer

Neo4j Meetup in Seattle - some observations

Some observations from a meetup in Seattle on graph databases and Neo4j

Ian Mercer
Ian Mercer

Updated Release of the Abodit State Machine

A hierarchical state machine for .NET

Ian Mercer
Ian Mercer

My first programme [sic]

At the risk of looking seriously old, here's something found on a paper tape

Ian Mercer
Ian Mercer

Building a better .NET State Machine

A state machine for .NET that I've released on Nuget

Ian Mercer
Ian Mercer

A simple state machine in C#

State machines are useful in many contexts but especially for home automation

Ian Mercer
Ian Mercer

MongoDB Map-Reduce - Hints and Tips

Ian Mercer
Ian Mercer

Why don't you trust your build system?

Ian Mercer
Ian Mercer

Elliott 803 - An Early Computer

Ian Mercer
Ian Mercer

Making a bootable Windows 7 USB Memory Stick

Here's how I made a bootable USB memory stick for Windows 7

Ian Mercer
Ian Mercer

Tip: getting the index in a foreeach statement

A tip on using LINQ's Select expression with an index

Ian Mercer
Ian Mercer

SQL Server - error: 18456, severity: 14, state: 38 - Incorrect Login

A rant about developers using the same message for different errors

Ian Mercer
Ian Mercer

WCF and the SYSTEM account

Namespace reservations and http.sys, my, oh my!

Ian Mercer
Ian Mercer

Mixed mode assembly errors after upgrade to .NET 4 Beta 2

Fixing this error was fairly simple

Ian Mercer
Ian Mercer

Shortened URLs should be treated like a Codec ...

Expanding URLs would help users decide whether or not to click a link

Ian Mercer
Ian Mercer

Tagging File Systems

Isn't it time we stopped knowing which drive our file is on?

Ian Mercer
Ian Mercer

A great site for developing and testing regular expressions

Just a link to a site I found useful

Ian Mercer
Ian Mercer

Introducing Jigsaw menus

A novel UI for menus that combines a breadcrumb and a menu in one visual metaphor

Ian Mercer
Ian Mercer

Fix for IE's overflow:hidden problem

Ian Mercer
Ian Mercer

A better Tail program for Windows

A comparison of tail programs for Windows

Ian Mercer
Ian Mercer

Measuring website browser performance

Found this great resource on website performance

Ian Mercer
Ian Mercer

Amazon Instance vs Dedicated Server comparison

Some benchmark performance for Amazon vs a dedicated server

Ian Mercer
Ian Mercer

Agile Software Development is Like Sailing

You cannot tack too often when sailing or you get nowhere. Agile is a bit like that.

Ian Mercer
Ian Mercer

Javascript error reporting

Sending client-side errors back to a server for analysis

Ian Mercer
Ian Mercer

AntiVirus Software is the Worst Software!

When your anti-virus software starts stealing your personal data, it's time to remove it!

Ian Mercer
Ian Mercer

ASP.NET Custom Validation

How to solve a problem encountered with custom validation in ASP.NET

Ian Mercer
Ian Mercer

Optimization Advice

Some advice on software optimization

Ian Mercer
Ian Mercer

Google Chart API

Ian Mercer
Ian Mercer

Cache optimized scanning of pairwise combinations of values

Using space-filling curves to optimize caching

Ian Mercer
Ian Mercer

Threading and User Interfaces

A rant about how few software programs get threading right

Ian Mercer
Ian Mercer

Take out the trash!

Why Windows shutdown takes so long

Ian Mercer
Ian Mercer

Dell upgrades - a pricey way to go

Ian Mercer
Ian Mercer

Programming mostly C#

Ian's advice on programming

Ian Mercer
Ian Mercer