Yes, we've thought a lot about using inotify, our first prototype was also using inotify.

- Our system needs to cope with millions of directories. Millions of directories for inotify mean a lot of structures in the kernel. For large numbers it can also mean gigabytes of ram. Add to that the mapping in userspace of file descriptors for rename operations.

- Using inotify would take a lot of time at startup to recurse into millions of subdirectories.

- inotify will not automatically watch new directories. You have to list all the files right after the creation of a directory and watch the subdirectories, etc. Not a problem at all, but way simpler with FUSE.

- If your system is not able to keep up with inotify events, you miss events because the kernel cannot buffer all the events. For us it's better to slow down the system but never miss an event.

- inotify is attached to the original filesystem. That means it's hard (not impossible) to handle loops in an active/active replication setup. Whereas with SFS, replication is done on straight to the original filesystem.

- If the inotify application crashes, you lose events because software keeps writing to the filesystem. If SFS crashes the mountpoint is unwritable by the application which reports an error and can switch to a different storage.

Some choices as you can read depends on the requirements, our requirements where not met by inotify.

There's a new thing (fanotify). LWN did a short series on different filesystem notification methods in July:

https://lwn.net/Articles/604686/

https://lwn.net/Articles/605128/

Unfortunately the fanotify article is yet to come!

(Having said that, I certainly appreciate your pain trying to use inotify to track write events on an entire filesystem)

fanotify is super limited, and not useful for this app. It's from 2009, but has seen little use.

AFAIK inotify isn't especially great for recursively watching a large directory.

Interesting, I haven't looked at the details and assumed it was similar to the osx FSEvents API. Maybe Linux has some catching up to do in this area?

We've done that as a workaround for old legacy web apps where nobody had originally thought about failover (and had spread data updates all over the filesystem, intermingled with code... It was not fun to untangle).It works, but we've always supplemented it with regular full filesystem rsync's.

Our cases have also had very limited semantics (uploads that adds new files or modify files, but no deletions ever) that made things easier.

This seems a lot cleaner...

Though these days we mostly use glusterfs and most of those old apps have thankfully been retired or cleaned up.

Glusterfs is for synchronous replication. Glusterfs asynchronous replication, at the time of writing SFS, only supported master-slave.

Yes, the point with bringing up gluster is that synchronous replication is the best solution for our use case (we have relatively low amount of filesystem writes, but need the files to be consistently replicated near-instantly).

But for applications where async is a good fit, your solution does definitively seem much nicer than the inotify + rsync stuff we were doing (we ended up with that - temporary - solution after writing off lsyncd, though I don't remember why).