News for hackers. Stuff that matters.
Re:S3 is files, but not a filesystem by macintux on Sunday March 10, 2024 @11:35AM (#39660006)
I don't know that you can characterize that as a "standard practice".

Maybe it's widespread, but I've not encountered it.

Re:S3 is files, but not a filesystem by belter on Sunday March 10, 2024 @12:08PM (#39660245)
Here is the architecture of Amazon Drive and the storage of metadata.

"AWS re:Invent 2014 | (ARC309) Building and Scaling Amazon Cloud Drive to Millions of Users" - https://youtu.be/R2pKtmhyNoA

And you can see the use here at correct time: https://youtu.be/R2pKtmhyNoA?t=546

Re:S3 is files, but not a filesystem by ianburrell on Sunday March 10, 2024 @01:38PM (#39660921)
That article is old. DynamoDB was used because of the old, weak consistency model of S3. Writes were atomic, but lists could return old results so needed consistent list of objects.

But in 2020, S3 changed to strong consistency model. There is no need to use DynamoDB now.

Re:S3 is files, but not a filesystem by belter on Sunday March 10, 2024 @08:45PM (#39663758)
The problem was not the eventual consistency model, was the speed of the object list.

"...Finding objects based on other attributes, however, requires doing a linear search using the LIST operation. Because each listing can return at most 1000 keys, it may require many requests before finding the object. Because of these additional requests, implementing attribute-based queries in S3 alone can be challenging..."

Re:S3 is files, but not a filesystem by orf on Sunday March 10, 2024 @08:52PM (#39663812)
It was both actually, but more for the listing issue. Netflix built a lot of tooling around this.

But yeah: things like filtering on tags or created at dates requires another approach.

Re:S3 is files, but not a filesystem by mr_toad on Monday March 11, 2024 @12:53AM (#39665016)
Hive stores metadata in a relational database. So does Snowflake.