Re:S3 is files, but not a filesystem
by type_Ben_struct on Sunday March 10, 2024 @04:03AM
(#39657517)
Tools like LucidLink and Weka go a way to making S3 even more of a “file system”. They break files into smaller chunks (S3 objects) which helps with partial writes, reads and performance. Alongside tiering of data from S3 to disk when needed for performance.
[ Parent
| on HN ]
I don’t know a whole lot about LucidLink but Weka basically uses S3 as a dataplane for their own file system.
Someone contributed an nbdkit S3 plugin which basically works the way you described. It uses numbered S3 chunks using the pattern "key/%16x", allowing the virtual disk to be updated. (https://libguestfs.org/nbdkit-S3-plugin.1.html https://gitlab.com/nbdkit/nbdkit/-/tree/master/plugins/S3)
The problem with these approaches is that the data is scrambled on the backend, so you can't access the files directly from S3 anymore. Instead you need an S3 gateway to convert from scrambled S3 to unscrambled S3. They rely on a separate database to reassemble the pieces back together again.