Interesting! We have been doing the same thing with HopsFS for a couple of years. Except, we only store the small files in our database (www.rondb.com) - RonDB is a recent fork of MySQL Cluster (NDBCluster). Very small files (<1KB) are stored in memory in RonDB, small files (typically <128KB) are stored in NVMe disks in RonDB, and other files in HopsFS (which now stores its blocks in object storage (S3, ABS).

We had a paper on it as ACM Middleware and it's open-source on github. Are you going to publish your solution?

(Discussed here on HN: https://news.ycombinator.com/item?id=25149154 )

Do you support atomic rename? Atomic rename on subtrees? Consistent directory listings?

That's a really interesting solution. Can you tell us more about it? Operating distributed blob storage systems is kind of fragile with every software i have yet tried.

FDB's Directory layer provides all you need to create and edit nested paths. What's left to develop is a file chunking and assembly part, and statistics if needed.

The only reason you need chunking is because FDB has very clearly defined limits in their documentation, and one of those limits is value size - it can't exceed 100kB, and should be kept below 10kB for best performance.

For statistics like folder byte count, you could use FDB's atomic increment operation.

Here's a good starting point (not mine): https://forums.foundationdb.org/t/object-store-on-foundation...

I'll be happy to answer specific question if you got any.

What kind of read/write ratio are you using? And would your solution work for a write-heavy workload?

Kafka has limits on the message size and i need a solution for storing large blobs (up to 10MB) at data ingestion on for a very short time until the job has been processed. So read/write ratio will be exactly 50% and there will be a high write load. Is FoundationDB capable for this specific task? Are there some knobs tuneable for acid? Perfect acid requirements would not really be needed, if everything is fsynced every second would be completely ok.

From own experience, I'd say FDB can handle it all, we've got 20% read 80 write during peak hours, and reverse: 80 read 20 write other time.

Without doing your own tests, here's per core numbers, from which you can extrapolate (e.g. via CPU mips) towards your own hardware: https://apple.github.io/foundationdb/performance.html#throug...

FDB is ACID as is shipped, you don't need to turn knobs to make it such. Toughest part is to figure out classes/roles of the system. Here are a couple of good starting points: https://nikita.melkozerov.dev/posts/2019/06/building-a-found... https://forums.foundationdb.org/t/roles-classes-matrix/1340/...

Kafka limits are customizable. 10MB is quite small. Kafka easily handles 100MB messages. If you need some temporal storage that works like a log then there's nothing better than kafka in terms of speed and scalability. But it's more about relatively long term storage. If you need to persist data for very short amount of time, why don't you use in-memory store? Do you have strict requirements around data loss? Something like Redis much just do the trick for out. With Redis cluster it even scales.