Re:Asynchronous filesystem replication with FUSE and rsync
by vidarh on Thursday December 11, 2014 @06:36PM
(#8737686)
Gluster does what you configure it to do, and unless you have a crazy setup it won't do what you think it will do.
For your type of setup, you'd combine the "distribute" and "replicate" translators. The "distribute" translator spreads files out over sub-volumes by hashed path. Unless you specifically enable "lookup-unhashed" it will not forward requests for files that are not found to any other sub-volumes than the one matching the hash.
The underlying volumes would then be "replicate" and you'd there decide how many replicas you want.
There are other cluster translators than distribute too, that provides more complex distribution methods, but there you do run into overheads with determining which subvolume holds the files.
You'd also usually layer the io-cache translator in (which you can add on both the clients and the servers, and which provides in memory caching), so while there are possible pathological cases, Gluster is pretty resilient.
Also Gluster will replicate directly between servers to heal, but read/writes using the fuse fs will have the client talk directly to each backend and write in parallel to each member of the replicate set for that file, so there normally is no separate replication unless a server is/was down or a disk has crashed: normally, once a write is reported as complete, the file is on all replicas.
But frankly almost regardless of file storage solution, your best bet is still to put Nginx or another suitable reverse proxy in front of the file servers, so you can e.g. cache the hottest requested files in faster/more expensive ways (memcached, or on SSDs)
For your type of setup, you'd combine the "distribute" and "replicate" translators. The "distribute" translator spreads files out over sub-volumes by hashed path. Unless you specifically enable "lookup-unhashed" it will not forward requests for files that are not found to any other sub-volumes than the one matching the hash.
The underlying volumes would then be "replicate" and you'd there decide how many replicas you want.
There are other cluster translators than distribute too, that provides more complex distribution methods, but there you do run into overheads with determining which subvolume holds the files.
You'd also usually layer the io-cache translator in (which you can add on both the clients and the servers, and which provides in memory caching), so while there are possible pathological cases, Gluster is pretty resilient.
Also Gluster will replicate directly between servers to heal, but read/writes using the fuse fs will have the client talk directly to each backend and write in parallel to each member of the replicate set for that file, so there normally is no separate replication unless a server is/was down or a disk has crashed: normally, once a write is reported as complete, the file is on all replicas.
But frankly almost regardless of file storage solution, your best bet is still to put Nginx or another suitable reverse proxy in front of the file servers, so you can e.g. cache the hottest requested files in faster/more expensive ways (memcached, or on SSDs)