My experience with SANs is that they are anything but consistent. Local storage is a better idea: fewer moving pieces to go wrong, fewer moving pieces to understand and debug, fewer possible sources of contention, and the latency is low.

SANs in a cloud environment optimize for the wrong thing. Servers by and large have a high uptime -- since their falling over is comparatively rare, this is simply a problem I've never had difficulty with. What I have had in spades, before I learned better, were database problems due to wild fluctuations in latency to the SAN.

It doesn't help that when SANs kick the bucket, they tend to affect a lot of things.

The context where SANs make sense, IMO, is when you've got a few servers which need to share stuff (VMs, or whatever). So, essentially everything can fit on one $10k 10GE switch. I've personally never screwed with anything >800TB, too.

Rather than "strictly local storage", I'd say "keep storage as local as possible", but there are absolutely times where keeping it in-chassis isn't optimal.

There are some using Ceph for the volume service who shouldn't be terribly expensive. Dreamhost for example.

Which providers are they? Do you have any experience of using them?