They are necessary because as soon as someone decides that S3 is a filesystem, they will look at the other cloud "filesystems," notice that S3 is cheaper than most of them, and then for some reason they will decide to run giant Hadoop fs stuff on it or mount a relational database on it or all other manner of stupidity. I guarantee you S3's customer-facing engineers are fielding multiple calls per week from customers who are angry that S3 isn't as fast as some real filesystem solution that the customer migrated from because S3 was cheaper.

When people decide that X is a filesystem, they try to use it like it's a local, POSIX filesystem, and that's terrible because it won't be immediately obvious why it's a stupid plan.

If a customer makes an IT decision as big as running Hadoop or RDBMS with S3 as storage ... but does not consult at least a Associate level AWS Certified architect (who are doke a dozen) for at least one day worth of advice which is probably a couple of hundred dollars at most ...

Can we really blame AWS?

I am sure none of official AWS documentations or examples show such an architecture.

----

Amazon EMR can run Hadoop and use Amazon S3 as storage via EMR FS.

"S3 mountpoints" are a feature specifically for workloads that need to see S3 as a file system.

For block storage workloads there is EBS and EFS and FSx that AWS heavily advertises.

*dime a dozen

(Apologies for typos. The "noprocrast" setting sometimes locks us out of HN right after submitting a comment. And it is now too late, not editable)

Yeah, it’s sort of funny how “POSIXish semantics” has become our definition of these things, when it’s just one kind of thing that’s been called a filesystem historically.

Fun experiment I made with my mum, building a storage independent dropbox like UI [1] for anything that implement this interface:

  type IBackend interface {
    Ls(path string) ([]os.FileInfo, error)
    Cat(path string) (io.ReadCloser, error)
    Mkdir(path string) error
    Rm(path string) error
    Mv(from string, to string) error
    Save(path string, file io.Reader) error
    Touch(path string) error
  }

My mum really couldn't care less about the posix semantic as soon as she can see the pictures of my kid which happen to be on S3

[1] https://github.com/mickael-kerjean/filestash

Reducing things to basically the interface you laid out is the point of 9p [1], and is what Plan 9’s UNIX-but-distributed design was built on top of. Same inventor as Go! If you haven’t dived down the Plan 9 rabbit hole yet, it’s a beautiful and haunting vision of how simple cloud computing could have been.

[1] https://9fans.github.io/plan9port/man/man9/intro.html

I think this interface is less interesting than the semantics behind it, particularly when it comes to concurrency: what happens when you delete a folder, and then try and create a file in that folder at the same time? What happens when you move a folder to a new location, and during that move, delete the new or old folders?

Like yes, for your mum's use case, with a single user, it's probably not all that important that you cover those edge cases, but every time I've built pseudo-filesystems on top of non-filesystem storage APIs, those sorts of semantic questions have been where all the problems have hidden. It's not particularly hard to implement the interface you've described, but it's very hard to do it in such a way that, for example, you never have dangling files that exist but aren't contained in any folder, or that you never have multiple files with the same path, and so on.

All those considerations are important when implementing the interface but the interface itself isn't invalidated by those concerns and can cope with those constraints fine.

Can S3 murder your wife like ReiserFS and Reiser4?

https://en.wikipedia.org/w/index.php?title=Comparison_of_fil...

Exactly, especially when the concept of filesystem really is defined before the whole internet scale becomes a thing or reality.

Maybe S3 isn't a filesystem according to this definition, but does it really matter to make it one? I doubt it. The Elastic Filesystem is also an AWS product, but you can't really work as one as you have locally, any folder over 20k files basically will timeout if you do a ls. Does it make EFS a filesystem or not?

The problem is once you let go of those semantics, a lot of software stops working if run against such a "filesystem". If you dilute the meaning of "filesystem" too much, it becomes less useful as a term.

https://en.wikipedia.org/wiki/Andrew_File_System was interesting, I'd actually love to see something similar re-implemented with modern ideas, but it's more of an direct-access archival system than a general-purpose filesystem[1], you can't just put files written by arbitrary software on it. It's a bit like NFS without locks&leases, but even less like a normal filesystem; only really good for files created once that "settle down" into effectively being read-only.

[1]: I wrote https://github.com/bazil/plop that is (unfortunately undocumented) content-addressed immutable file storage over object storage, used in conjunction with a git repo with symlinks to it to manage the "naming layer". See https://bazil.org/doc/ for background, plop is basically a simplification of the ideas to get to working code easier. Site hasn't been updated in almost a decade, wow. It's in everyday use though!