Date Tags devops

Blue/Green deployments are ever popular in the DevOps world these day. Do a quick Google search and you will find ample blogposts telling you how people do them and why they're a good thing.

You'll likely see references to Continuous Delivery, Jez Humble, Dan North, Martin Fowler, Julian Simpson among others who have helped push these ideas into the mainstream in recent years.

In fact, if you're ever lucky enough to hear one of these guys talk, you will hear a tale about the first place the method was put into practice on a UK ISP's ADSL provisoning system around 2004/2005.

I was fortunate enough to be the Ops/Sysadmin at the ISP at the time, and was also fortunate enough to be the point man for the build team that included Jez Humble et al (thanks to my boss throwing me into the project).

I was also the one who said that the deployment tool should be written in Bash, something which is often mentioned in talks about Continuous Delivery and always gains a few chuckles.

That decision may seem like madness today - given the ever growing suite of deployment tools out there - however, back then we didn't really have those tools, what we had was a team of sysadmins where the lowest common denominator skill was shell scripting.

This is why shell was chosen. After all, when Jez et al had left the building, we would need to support deployments from that point on. Using Bash meant that we could be pretty certain everyone could understand - and hack if necessary - the deployment tool, something we did do over the following years it was in use.

Conan the Deployer and Blue/Green

Using shell scripts gave birth to the first and original Conan the Deployer . This post is a bit more detail on what it did in its use of the blue/green deployment technique just over a decade ago, and what the very first blue/green deployment to production looked like.

The blue/green deployment strategy is a pretty simple one, and to be honest it's pretty much common sense to me. You have production running in a colour, let's say blue. You build a new environment along side it as a "shadow" and you label it green.

Once you're happy that green is deployed and ready, you flip over to it and you're done. This reduces any potential outage to during the moment of cutover. What's more, if you have a problem you can flip back so you get almost instant rollback. This makes change management happy - as does marrying the Change Manager but that is a different story!

Here's what the aging old system looked like. There were three Solaris 8 servers in total, each running Apache 1.3 and Weblogic 8, and the database was running on Sybase ASE12 (IIRC).

So we had our live url, and then on one of the servers we had a smoketest url. When you ran Conan, the collection of functions and for loops would do the following:

  1. Check out the relevant release from CVS (later SVN).
  2. Build a Weblogic 8 domain locally with templated config using the port numbering schema for the colour you were deploying..
  3. Build an empty "shadow" database containing the latest schema for the release.
  4. Push the Weblogic domain (as a tarball) to the target hosts and start it up.
  5. Point the smoketest url at the new "shadow" release in green.

At this point we had a full stack of the application pointing at an empty database and QA could run whatever smoke tests they needed via the smoketest url. Tests usually involved generating a faked order.

We achieved this thanks to the implementation of a mock server running in WebLogic that would mimic responses from the external SOAP endpoints the system had to interact with. Once smoke tests were passed we were ready to let Conan cutover to the new environment and release.

Cutover was a ridiculously simple, and frankly elegant process which can be summed up in the one liner below. Basically, Conan ran a for loop to the three servers and did something a little like this:

cd /etc/apache/; rm httpd.conf; ln -s httpd-green.conf httpd.conf; /etc/init.d/apache reload

That's it.

Apache's config file was a symlink to either a blue or green config. Change the symlink and HUP apache. Job done.

Sure there was a tiny possibility that a user could lose their session, but this system was not operating at scale so the risk was considered acceptable.

At a stroke we went from deployments spanning entire weekends, to doing them on Wednesday lunchtimes - we were operating under Sarbannes-Oxley so change windows were limited - all thanks to a handful of shell scripts that the build guys had worked on for non-production, and I helped make suitable for production.

Everyone was happy. Deployment was now uniform. Anyone could do it (assuming they had the level of access). And that is the story of what I believe was the first Blue/Green deployment into production and the set-up behind it.

Obviously today, blue/green set-ups come in many shapes and size, be they custom images in the Cloud, flipping DNS or load balancers.

Conan the Deployer lived on for many years at the ISP. It was also forked and re-implemented for newer versions of Weblogic and larger systems like ESB. As too was using blue/green everywhere.

Epilogue

Sadly the ISP's customer base in the UK was eventually sold on, subsumed into a corporation that had grown through user acqusition, and was then mothballed in favour of a large-scale monolithic provisioning system that could only have deployments with total outages for 8 hours, but I had left by the time things took a giant leap backwards.



Comments

comments powered by Disqus