Also, does Ceph distribute it's objects so that two drives don't contain the same set of objects? I.E. it's probabilistically impossible that a number of drives going down in a large array can wipe out all the copies of the object?
Yes, you create a CRUSH map which lets you define a hierarchical list of bucket types that reflect your failure domains (host, chassis, rack, room, datacenter, etc).
http://ceph.com/docs/master/rados/operations/crush-map/#crus...
And yes, in the clusters I've built I've always calculated what the chances are that X simultaneous drive failures will take out any data, and it's always been astronomically low.
I've mostly seen people use triplicate pools. For semi-warm storage I've been testing erasure coded pools, with a triplicate cache tier on top, and had good experiences on my test cluster.
I've never used Ceph in production, but it doesn't seem unreasonable to expect the sysadmin to keep clock skew in check.
Ceph object-store known as RADOS is production ready and supported by Red Hat [1], in addition the block-device service known as RBD which is an access method to RADOS is also production ready and supported. RBD provides many SAN like features like snapshots, cloning, and soon mirroring. The CephFS (POSIX filesystem access method) is not yet a supported product but is used by several sites in production. Ceph also provides S3/Swift object-store gateway (radosgw) so existing applications can access the object store with a compatible API.
Ceph provides unified storage: providing object, block and filesystem access from a single cluster, so Ceph is the more general purpose technology. Ceph MDS provides metadata service for the CephFS POSIX-filesystem but is not needed if you're not using that feature. For development and testing Ceph will run on a single VM, or more realistically on 3 small VMs so the cluster characteristics can be explored.
[1] alternatively you can buy a production grade fully supported appliance product from Fujitsu based on Ceph known as ETERNUS CD10000. For most deployments Ceph runs well on commodity hardware and the Ceph community provides excellent support too.
EDIT: And I'm not suggestion it isn't badass, just that the idea Ceph as a first suggestion for general purpose RAID replacement is probably not a good idea just yet, and that there's an alternative.
What John is not talking about is the Object Store (S3 compatible) or RBD (rados block device, EBS like block storage). There's countless people using both Object Store and RBD in production today. Both of these are the most deployed solutions for OpenStack (for storage or object storage). For example the Bloomberg folks use in production.
We use CephFS, and contributed to it earlier, and helped find a lot of bugs in the kernel and the MDS. Last time we uncounted a bug was over a year ago. We currently only use one MDS with a hot spare (backup node actively replaying the log).
What's really missing today from CephFS is the fsck and restore utilities and I believe people are working on them as we speak.
DreamHost would beg to differ. They've been using it for their DreamObject[1] for few years. Granted that they have have better visibility earlier because Sage Weil, but I've seen Ceph deployment for production in the wild since mid-2014.