One of the rather nifty things with AWS is the ability to create autoscaling groups to handle surges in traffic when you're operating at scale. However, you don't have to just use auto-scaling to make the number of instances for an application larger, you can also use it to maintain a set number and effecitvely self-heal the system on failure.
If you create an autoscaling group with a min/max setting that is equal then you can effectively maintain that size, however, there is a problem if you want to use EBS volumes. Launch configuration for the group can define the volumes you want but there is no means in the current set-up to attach the volumes on boot, what's more, how does the system know which volume to attach where?
This was a problem I encountered when I was invetsigating using a clustered InfluxDB as a storage engine for Graphite. I wanted to be able to have an autoscaling group that would span all availability zones in a given region, where the data for influx would be stored on EBS and would, in the event of failure, re-attach itself when Amazon terminated and relaunched the instance.
Using Instance-Profile roles to allow access to the API, I knocked up the following dirty little script that could run at boot (using cloud-init or the standard user-data in Amazon), talk to the API and say "can I have an EBS volume in my AZ that has this tag and is in an available state please?"
At this point, all I had to do was create my required EBS volumes in advance in each AZ, apply a tag to them with a value that was consistent through, e.g.
usage: attach_volume.py [-h] [--tag TAG] --value VALUE --attach_as ATTACH_AS Attach EBS Volume optional arguments: -h, --help show this help message and exit --tag TAG Tag key, defaults to Name --value VALUE The tag value to search for --attach_as ATTACH_AS device path e.g. /dev/xvdb
The script is avilable in my
aws_stuff repo on GitHub