We recently upgraded one of our etcd clusters from v2 to v3 and it turned out that etcd v3 uses significantly more IOPS than v2, using EBS burst credit during periods with normal load. Our solution was to increase the EBS volume sizes from 16GB to 50GB (100 IOPS -> 150 IOPS [1]). Our costs went from $1.6 per volume to $5, instead of $11.75 for io1, for a comparable performance for this particular application.
1: http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/EBSVolume...
It's a trade you must account for when choosing cloud providers, nothing is free. For now it's still cheaper to pay for servers than spend time setting things up ourselves, but we're always measuring cost x benefits and thinking if we need to leave aws. The time is coming, but for now it's still working for us.
shockingly, tons of people don't know about this. Sure, there are use cases where it probably isn't cost effective to do this, but my default policy towards using gp2 is "will this server write to disk even a moderate amount? if so, its get a 1TB volume." Especially for something like a node in a k8s cluster where your workload is non-deterministic, this should be the default.
That's my takeaway as well. If you can track and alert on it, you can provision whatever you want and only buy the expensive io1 when it makes sense.
gp2 disks get 3 iops per gig "free", bursting up to 3k. (They don't burst after 1 tb, because your baseline performance is higher than the burst rate.) io1 is 25% more expensive per-gb, and you pay by the IOPS on top of that.
A 175gb gp2 disk will give you 525 IOPS baseline, at $17.50 a month. I'm guessing his volume is about 40gb, doing math backwards from his bottlenecked IOPS; a 40gb io1 with 500 IOPS will cost you $83.75 a month! And on top of that, AWS will cap you hard at that 500 IOPS; the gp2 can still burst if needed.
I know of two general cases where io1 disks are worthwhile: if you need more than 10k IOPS, or if you have very high IO requirements but very stable disk space usage (e.g. high-performance RRDs). Crack open Excel and do the math; it's worth the five minutes to check.
(Also, your burst balance is available as a Cloudwatch metric, as I recall! Set alarms on that shit!)