Redis is as reliable as the storage you persist it to. If you're running Redis right, it's very reliable. Not S3 reliable, though. But if you need S3 reliable, you would turn to something else.

I expect that most folks looking at this are doing it because it means:

1. Effectively unbounded storage

2. It's fast

3. It's pretty darn cheap

4. You can scale it horizontally in a way that's challenging to scale other filesystems

5. All the components are pretty easy to set up. Many folks are probably already running S3 and Redis.

Redis isn't durable unless you drastically reduce the performance.

Filesystems are pretty much by definition durable.

Enabling the WAL doesn't make Redis slow. It's slower than the default, but it's still exceptionally fast.

> Filesystems are pretty much by definition durable.

Where do you think Redis persists its data to

Redis using the WAL for durable data can only be as fast as you can fsync on every write, just like every other database. The time to fsync is going to dominate the write path for Redis, and you'll sacrifice almost all of Redis's speed to achieve it. If you're running Redis in always fsync mode it I'd suggest there are better storage systems to use.

> Where do you think Redis persists its data to

This is sort of the point, Redis isn't for persistence. I mean it can, you can use the WAL, it's handy for not having to recompute when you relaunch Redis etc, but Redis is fundamentally an in-memory system and should be treated as such. It is designed for use-cases that necessitate in-memory performance, and that don't require durability.

> 4. You can scale it horizontally in a way that's challenging to scale other filesystems

Easy to scale on RDS, along with everything else. But there’s no Kubernetes operator. Is there a better measure of “easy” or “challenging?” IMO no. Perhaps I am spoiled by CNPG.

> Redis is as reliable as the storage you persist it to.

For a single node, if you tank performance by changing the configuration, sure. Otherwise, no, not really.

I don't get why you'd want a file system that isn't durable, but to each their own.

With the wal redis runs with perfectly reasonable performance. Of course you're not going to have the performance of an in-memory only DB if you're flushing to disk on every write.

There's no vacuuming, there's no need for indexing. You can see the time complexity of most operations. Key-value operations are mostly O(1). You'll never get that kind of performance with other databases because they intentionally don't give you that granularity.

The metadata of the filesystem isn't the performance bottleneck in most cases.

> With the wal redis runs with perfectly reasonable performance. Of course you're not going to have the performance of an in-memory only DB if you're flushing to disk on every write.

But that's not what gets benchmarked against.

> You'll never get that kind of performance with other databases because they intentionally don't give you that granularity.

Okay but you also won't get horizontal scaling while maintaining consistency in Redis.

> The metadata of the filesystem isn't the performance bottleneck in most cases.

Okay, but they do show that other metadata stores are 2-4x slower than Redis, so it would be great to see whole-system benchmarks that use those.

I think they should replace Redis with Valkey or even better use rocksdb.

But how would RocksDB work with S3? It needs support for append that generic S3 buckets do not provide and for checkpoints and backup it assumes the support for hardlinks that S3 does not have at all.

SeaweedFS has a RocksDB metadata backend for instance.

It says MySQL can be used instead of Redis for the metadata

yes, support more 10 options, include redis, SQL-like DB, TiKV, FoundationDB, and more. see here -> https://juicefs.com/docs/community/databases_for_metadata

We developed Object Mount (formerly cunoFS) (https://www.storj.io/object-mount?hn=1) specifically to not rely on any metadata storage other than S3 AND preserve 1:1 mapping of objects to files AND support for POSIX. We have a direct mode that uses LD_PRELOAD to keep everything in userspace so no FUSE overhead.

This approach isn't right for every use case and juice might be a better fit for this sort of 'direct block store', but wanted to include it here for folks that might want something like Juice but without having to maintain a metadata store.

(Disclosure: I work at Storj that develops Object Mount)

Pretty cool

Is this open source? I am a happy Storj customer, would love to use it if it's open source.

Not open source. What would be your use case?

I am currently looking for a way to take a legacy application that uses the filesystem as it's database and needs to support locking (flock) on FreeBSD and scale it horizontally (right now we only scale vertically and rebuilding from a corrupted FS and/or re-pulling our, backup, data from S3 takes too long if we lose a machine). We investigated NFS but the FreeBSD NFS performance was 1/10th or worse than on Linux and switching to Linux is not in the cards for us right now.

Does Object Mount support file locks (flock specifically) and FreeBSD? I see some mention of FreeBSD but I can't find anything on locking.

For context, we are working with a large number of small (<10KB if not <4KB) files normally.

No BSD support unfortunately. Just Linux, Mac, windows.

According to this link:

https://vonng.com/en/pg/pgfs/

> The magic is: JuiceFS also supports using PostgreSQL as both metadata and object data storage backend! This means you only need to change JuiceFS’s backend to an existing PostgreSQL instance to get a database-based “filesystem.”

Sounds ideal for your kind of situation where a filesystem is being abused as a database :-)

JuiceFS metadata engine comparison -> https://juicefs.com/docs/community/metadata_engines_benchmar...

Thanks, good to see these. Looks like these only show metadata operations though. But I guess that's probably enough to extrapolate that the system is going to be roughly 2-4x slower with other metadata stores.