We run hundreds of GlusterFS clusters in production on EC2. We're currently on 3.0 and in the process of fully migrating to 3.4 (and maybe 3.5 one day).

Our primary use case for Gluster is in serving persistent filesystems for Drupal. Our customers store potentially millions of files on their GlusterFS clusters.

We've built a number of tools/processes to help protect Gluster against failures in EC2 (for instance fencing network traffic at the iptables layer to help protect GlusterFS clients from hanging talking to down nodes), as well as to help our team perform common tasks (resizing clusters, moving customers from cluster to cluster, etc.). We haven't necessarily hit blocker issues recovering from underlying hardware fails, but our team is definitely very experienced with many possible failure modes.

Overall GlusterFS has been very reliable over the years and our research has shown it is the best option out there for when our customers can't use something such as S3 directly.

If you want more details or would love to hack on a 8000+ node EC2 cluster running things such as GlusterFS feel free to ping me.

At 8k+ nodes, bare metal would likely be much cheaper. Why would you possibly want to host this on EC2?!

Disclaimer, I've been at companies with thousands of servers for my past 3 jobs. I've never once used any "cloud" service other than Linode for my personal VPS and ganeti for the Oregon open source lab VM donated to the gnome foundation (I'm a sysadmin alum for gnome.org)

Just out of interest why don't you use S3 for this? Amazon provides a few options for scalable storage, is it cheaper to roll your own ontop of EC2?

Many of our customers do use S3 and we make use of S3 extensively ourselves. However, Drupal often expects to operate on a POSIX compatible filesystem. Drupal 7 does support PHP file streams which can be configured to use S3, but not every Drupal module follows the best practices. Plus, we support every flavor of Drupal under the sun (including custom code).

All of our enterprise customers receive a highly available setup running on multiple nodes--thus, we have the need for a persistent filesystem attached to multiple EC2 instances. We utilize GlusterFS to ensure all of our clients have the filesystem capabilities their apps may need.

Basically, they need EBS without the single-instance (writable) mount limit, if they're using it to sync Drupal installations (php files, temp folders, etc.) across multiple web servers. S3 isn't really intended for that kind of workload (it would be possible to use something like s3fs, but I don't think that will give you a POSIX-compatible file system which is required for Drupal, and it's probably too slow as well (lots of small files, etc.))

We run it in some 2-node configurations with 4-6 bricks/node (Gluster is configured as distribute+replicate). Replacing disks isn't a big deal, there's a replace-brick command which migrates the data away from a brick to a new one, or for hard fails you can just drop in a new disk, do replace-brick and force a heal to re-replicate data. A node going down and coming back is straightforward, the self-heal-daemon and disk activity get the "missed" changes re-replicated (or force it via heal) -- everything keeps working in the meantime. Native clients ignore "down" nodes without needing intervention (for NFS clients use a shared IP and fail that over to any "up" gluster node).

Performance gets better with every release, but the speed of `ls` (Gluster uses `stat()` to trigger replication/checking) is still woeful at times. 3.5 will hopefully make that a lot better. In general, Gluster is not the right answer if you have lots of small files.

I used a very early version of GlusterFS on a 5 node cluster and never lost any data (about 32T worth, quite a bit at the time). What I liked most about it was that it produced storage nodes that could be inspected with the regular toolset, even if something would go down (which it never did) it would definitely have been possible to recover by looking at the crashed storage.

Having the 'bricks' use the regular filesystems was a very smart decision, it means you don't end up in binary hell when things go south.

Split brain was one of the weaker points back in the days, I am not current with the software so no idea if and how much this has improved, someone with more recent experience would be a better guide there.

Sadly, split brain is still a bit of a sore spot. It probably will be as long as GlusterFS continues to use the "fan out" replication approach, though past and future enhancements around quorum should help too. On the other hand, it's great to get such positive feedback about the "everything is regular files" doctrine. It has certainly been a bit painful sometimes, especially in terms of performance, but the feeling has always been that the ops advantages are worth it. Nice to see someone else agrees.

(Disclaimer: I'm a GlusterFS developer)

Hah, cool. Is 'dtor' still with the team? If so please say hi to him.

Well, we used it for storage backend for VMs few years back.

For some reasons it usually failed on high load, resulting in bad split-brains, extremely slow IO and loss of data. Very bad. We contacted Red Hat regarding that and got comments that gluster was not VM ready yet. We ditched gluster and went for plane proven old iSCSI.

This. We had the same failure mode, on a cluster with dozens of nodes and thousands of VMs.

http://thr3ads.net/gluster-users/2011/06/480298-Enomaly-user...

This bug (and others) would happen when moving files and/or folders over top of existing ones, ie:

  mkdir -p foo/bar/baz && \

  mkdir -p foo/bar/tmp && \

  touch foo/bar/baz/file foo/bar/tmp/file && \

  mv foo/bar/tmp/file foo/bar/baz/ && \

  rm -rf foo/bar/tmp

This was a method to replace an existing lockfile. Mercurial uses some similar code. The expected behavior would be that foo/bar/tmp/file replaced /foo/bar/baz/file. Instead, the outcome was that there was a race in Gluster where it got confused about which version of 'file' was correct, and it would end up with a split brain between the two nodes. This would be exacerbated by a node failure, but didn't always require one. Heavy load seemed to make the failure more likely. We couldn't replicate the same bug moving files in the same folder, it was the subfolders in the same gluster fs that seemed to cause the issue. The frustrating part was how gluster pointed fingers at the file operation being incompatible, despite advertising a posix compatible filesystem. Apparently the bug is fixed, but we moved off gluster, never to return.

Also, filesystems are hard, I get it, so no hard feelings :)

I've used it on production in EC2. The setup essentially replicated the underlying files to at least two nodes so if a host failed then the volume was still entirely available automatically, and when the dead host recovered it would replicate on read. It was great for mostly read workloads, but sluggish when the workload involves a lot of writes due to replication costs.

I used GlusterFS a couple of years ago (2011) for a hosting project, we had about 300 or so medium-large PHP sites based on top of a GlusterFS cluster. It ran beautifully but the pain points where it fell down was small file performance and its configuration system (I think this has been made much nicer in more recent versions).

I had a similar experience trying to build a highly-available wordpress infrastructure a few years back. The terrible latency of gluster completely killed any chance of getting decent web performance out of it. Ended up implementing OCFS2 on DRBD, much better.

Were you serving the PHP files off of Gluster? Or just the static files?

I had a client running PHP from a gluster storage, split across two nodes. Performance was horrific, although some of the sites were improved via Varnish we eventually had to move to local copies of files & rsync. Not pleasant.

Gluster is nice, but you have to accept the fact that it is slow, and upgrades are painful. (We had issues where clients and servers had to be updated at the same time, to avoid interoperability issues.)

Using it in production (v3.4.0), but haven't had a critical fail yet.

The fact that I keep my "bricks" on top of RAID(6) also helped with that.

We used to use gluster in production but had similar problems to raggiskula and are currently in the process of moving away from it.