News for hackers. Stuff that matters.
Re:S3 is files, but not a filesystem by type_Ben_struct on Sunday March 10, 2024 @04:03AM (#39657517)
Tools like LucidLink and Weka go a way to making S3 even more of a “file system”. They break files into smaller chunks (S3 objects) which helps with partial writes, reads and performance. Alongside tiering of data from S3 to disk when needed for performance.
Re:S3 is files, but not a filesystem by hnlmorg on Sunday March 10, 2024 @04:44AM (#39657659)
I don’t know a whole lot about LucidLink but Weka basically uses S3 as a dataplane for their own file system.
Re:S3 is files, but not a filesystem by rwmj on Sunday March 10, 2024 @07:10AM (#39658267)
Someone contributed an nbdkit S3 plugin which basically works the way you described. It uses numbered S3 chunks using the pattern "key/%16x", allowing the virtual disk to be updated. (https://libguestfs.org/nbdkit-S3-plugin.1.html https://gitlab.com/nbdkit/nbdkit/-/tree/master/plugins/S3)
Re:S3 is files, but not a filesystem by cuno on Sunday March 10, 2024 @05:29PM (#39662668)
The problem with these approaches is that the data is scrambled on the backend, so you can't access the files directly from S3 anymore. Instead you need an S3 gateway to convert from scrambled S3 to unscrambled S3. They rely on a separate database to reassemble the pieces back together again.