News for hackers. Stuff that matters.

A comment on S3 Strong Consistency. Parent comment

Re:S3 Strong Consistency by roseway4 on Tuesday December 01, 2020 @07:29PM (#25271913)
Yes. Writing objects from EMR Spark to S3 buckets in a different account. Immediately setting ACLs on the new objects fail as the objects are not yet “there”. Using EMRFS Consistent View was previously the obvious solution here, but it adds significant complexity.
[ Parent | on HN ]
Re:S3 Strong Consistency by enw on Tuesday December 01, 2020 @09:39PM (#25272689)
EMRFS consistent view is a mess. Changes to S3 files are likely to break EMRFS, the metadata needs to be kept in sync (otherwise the table will keep increasing forever) with tools that are a pain to install, and when I used it to run Hive queries from a script, it didn't work.
[ Parent | on HN ]
Re:S3 Strong Consistency by jamesblonde on Wednesday December 02, 2020 @12:38AM (#25273577)
This news is the death of EMRFS. The only thing it has left is fast listings of "directories" using DynamoDB.
[ Parent | on HN ]