Up until recently, and by virtue of using AWS for quite a while, I would always err on the side of caution when using EBS in production and punt for Provision IOPs (PIOPs) volumes. Obviously choosing the right number of PIOPs is crucial, although in fairness I think it might often be done by the "finger in the air" method.
I know this is obviously bad, but I imagine it happens a lot. The question is, do you really need PIOPS at all?
If you've been using AWS for any significant length of time you may think the answer is "Hell Yes!", and this is likely because you can remember some terrible time when standard EBS without PIOPs was so poor you couldn't rely on it to guarantee you any level of comfort. People tend to be the sum of their experiences after all, and if you've got an EBS war story in your back pocket then it's game over. In fact, you might be flipping the table and thinking I'm nuts for even posing the question.
Here's a confession, until recently I was very much in that, "standard EBS sucks balls" camp, but then I realised that I was wrong. Given my propensity for arrogance this acknowledgement was a rare occurrence obviously! That's pretty much why I've done this post, because if you're using EBS already (with PIOPs) or are perhaps planning too, you should probably consider the following:
Things to remember about EBS
EBS is a service not a disk. It may look like a disk, act like a disk, and walk like a disk, but it doesn't quack! Think of EBS more as an abstracted, highly available durable and performing storage service. It's not a SAN like you buy for your datacentre. If you think of it in those terms you will be cursing your lack of preparedness in the event of a failure.
Whilst EBS offers persistent, you should try to architect your use cases with the assumption that it isn't. This might sound counter-intuitive but you should think about how you design your systems so you can take a hit on EBS in an availability zone at the very least. Regular snapshots and recovery is beyond the scope of this blogpost obviously.
Don't use the magnetic option unless you know why you want too. i.e. you intend to store infrequently accessed data. You should use GP2 SSD volumes by default.
Understand how IOPs actually work with GP2 volumes as this will guide you in whether you really need to ask for PIOPs and pay the extra money.
How GP2 Volumes work
Standard GP2 volumes provide a baseline level of IOPs based on the size of the volume, calculated on the basis of 3 IOPs/per GB provisioned, and they can burst to 3000 IOPs. What this means in practice is that a 100GB volume will have a baseline performance throughput rate of 300 IOPs plus the bursting capability up to 3000.
Bursting is based on a bucket system that is refilled at the same rate as the baseline calculation, i.e it refills at 3 IOPs/per GB provisioned, and the bucket starts with 5.4 million IOPs on all volumes.
I'm not going to do the maths here, instead I will just say that for most applications, you're unlikely to ever actually exhaust your bucket, i.e. burst for long enough that you empty the bucket and cannot refill it fast enough. It's worth noting though that if you provisioned a 1TB volume you would never exhaust the bucket as your baseline rate would be 3000 IOPS.
You might be, at this point, wondering what exactly the point of PIOPs is if the standard volumes have this sort of performance. Essentially the difference comes down to performance consistency.
Offically, standard GP2 volumes have a performance consistency of 99%, compared to 99.999% when you use PIOPs. However, according to the EBS team at AWS, these SLAs have been far exceeded since the introduction of standard GP2.
So the reality is you'll probably get the offical PIOPs level of consistency with standard volumes - at a fraction of the cost - albeit without the guarantee.
Obviously deciding whether you want to take this risk is a down to you, however, as a real world example, in recent weeks at work we moved our Cassandra cluster with PIOPs to standard GP2 after we checked our metrics and realised the the PIOPs baseline wasn't adding any value other than to Amazon's own wallet. Obviously our set-up allowed us to take the trade-off and save significant amounts of money.
What the hell does all this mean?
I guess what I'm trying to say is don't go rushing in with PIOPs unless you really have a use case for it. The cost overhead may in fact be totally unnecessary. What's more, don't rely on your previous experiences of EBS from a few years ago, it's really not that bad any more. As a result of changing over to GP2 standard you could end up with faster volumes for a fraction of the cost.
Disclaimer: If you choose to use GP2 over PIOPs and it doesn't work then it's not my fault! :-)