News for hackers. Stuff that matters.
Re:S3 is files, but not a filesystem by orf on Sunday March 10, 2024 @08:08AM (#39658507)
> And listing files is slow. While the joy of Amazon S3 is that you can read and write at extremely, extremely, high bandwidths, listing out what is there is much much slower. Slower than a slow local filesystem

This misses something critical. Yes, s3 has fast reading and writing, but that’s not really what makes it useful.

What makes it useful is listing. In an unversioned bucket (or one with no delete markers), listing any given prefix is essentially constant time: I can take any given string, in a bucket with 100 billion objects, and say “give me the next 1000 keys alphabetically that come after this random string”.

What’s more, using “/“ as a delimiter is just the default - you can use any character you want and get a set of common prefixes.

This is super powerful, and it’s the thing that lets you partition your data in various ways, using whatever identifiers you need, without worrying about performance.

If listing was just “slow”, couldn’t list on file prefixes and got slower proportional to the number of keys (I.e a traditional unix file system), then it wouldn’t be useful at all.

Re:S3 is files, but not a filesystem by adrian_b on Sunday March 10, 2024 @08:37AM (#39658595)
Since 30 years ago (starting with XFS in 1993, which was inspired by HPFS) all the good UNIX file systems implement the directories as some kind of B trees.

Therefore they do not get slower proportional to the number of entries and listing based on file prefixes is extremely fast.

Re:S3 is files, but not a filesystem by nh2 on Sunday March 10, 2024 @09:18AM (#39658820)
> listing based on file prefixes is extremely fast

This functionality does not exist to my knowledge.

ext4 and XFS return directory entries in pseudo-random order (due to hashing), not lexicographically.

For an example, see e.g. https://righteousit.wordpress.com/2022/01/13/xfs-part-6-btre...

If you know a way to return lexicographical order directly from the file system, without the need to sort, please link it.

Re:S3 is files, but not a filesystem by kbolino on Sunday March 10, 2024 @11:25AM (#39659906)
Resolving random file system paths still gets slower proportional to their depth, which is not the case for S3, where the prefix is on the entire object key and not just the "basename" part of it, like in a filesystem.
Re:S3 is files, but not a filesystem by orf on Sunday March 10, 2024 @08:44AM (#39658620)
Yes they do. What APIs does Linux offer that allows you to list a directories contents alphabetically starting at a specific filename in constant time? You have to iterate the directory contents.

You can maybe use “d_off” with readdir in some way, but that’s specific to the filesystem. There’s no portable way to do this with POSIX.

Regardless of if you can do it with a single directory, you can’t do it for all files recursively under a given prefix. You can’t just ignore directories, or say that “for this list request, ‘-‘ is my directory separator”.

The use of b-trees in file systems is completely beside the point.

Re:S3 is files, but not a filesystem by adrian_b on Sunday March 10, 2024 @08:50AM (#39658646)
The POSIX API is indeed even older, so it is not helpful.

But as you say, there are filesystem-specific methods or operating-system specific methods to reach the true performance of the filesystem.

It is likely that for maximum performance one would have to write custom directory search functions using directly the Linux syscalls, instead of using the standard libc functions, but I would rather do that instead of paying for S3 or something like it.

Re:S3 is files, but not a filesystem by orf on Sunday March 10, 2024 @08:54AM (#39658669)
Yes. You could also just use a SQLite table with two columns (path, contents), then just query that. Or do any number of other things.

The question isn’t if it’s possible, because of course it is, the question is if it’s portable and well supported with the POSIX interface. Because if it’s not, then…

Re:S3 is files, but not a filesystem by anamexis on Sunday March 10, 2024 @09:00AM (#39658697)
> The question isn’t if it’s possible, because of course it is, the question is if it’s portable and well supported with the POSIX interface. Because if it’s not, then…

Where did this goalpost come from? S3 is not portable or POSIX compliant.

Re:S3 is files, but not a filesystem by orf on Sunday March 10, 2024 @09:09AM (#39658743)
From the article we're commenting on, which is comparing the interface of S3 to the POSIX interface. Not any given filesystem + platform specific interface.
Re:S3 is files, but not a filesystem by anamexis on Sunday March 10, 2024 @09:16AM (#39658809)
The article does not mention POSIX, or anything about listing files, at all.
Re:S3 is files, but not a filesystem by orf on Sunday March 10, 2024 @10:06AM (#39659182)
It mistakenly mentions UNIX whilst referencing the POSIX filesystem API, and I literally quoted where it talks about listing in my original comment.
Re:S3 is files, but not a filesystem by zaphar on Sunday March 10, 2024 @09:39AM (#39658954)
The article starts out by making a comparison between the posix api filesystem calls and S3's api. The context is very much a comparison between those two api surface areas.
Re:S3 is files, but not a filesystem by justincormack on Sunday March 10, 2024 @09:09AM (#39658745)
There are no specific syscalls that you can use for this. The libc functions and the syscalls are extremely similar.
Re:S3 is files, but not a filesystem by jacobsimon on Sunday March 10, 2024 @08:39AM (#39658602)
What is it about S3 that enables this speed, and why can’t traditional Unix file systems do the same?
Re:S3 is files, but not a filesystem by orf on Sunday March 10, 2024 @08:52AM (#39658656)
S3 doesn’t have directories, it could be thought of a flat + sorted list of keys.

UNIX (and all operating systems) differentiate between a file and a directory. To list the contents of a directory, you need to make an explicit call. That call might return files or directories.

So to list all files recursively, you need to list, sort, check if an entry is a directory, recurse”. This isn’t great.

Re:S3 is files, but not a filesystem by bradleyjg on Sunday March 10, 2024 @09:03AM (#39658707)
Code written against s3 is not portable either. It doesn’t support azure or gcp, much less some random proprietary cloud.
Re:S3 is files, but not a filesystem by cuno on Sunday March 10, 2024 @10:36AM (#39659452)
Actually we've found it's often much worse than that. Code written against AWS S3 using the AWS SDK often doesn't work on a great many "S3-compatible" vendors (including on-prem versions). Although there's documentation on S3, it's vague in many ways, and the AWS SDKs rely on actual AWS behaviour. We've had to deal with a lot of commercial and cloud vendors that subtly break things. This includes giant public cloud companies. In one case a giant vendor only failed at high loads, making it appear to "work" until it didn't, because its backoff response was not what the AWS SDK expected. It's been a headache that we've had to deal for cunoFS, as well as making it work with GCP and Azure. At the big HPC conference Supercomputing 2023, when we mentioned supporting "S3 compatible" systems, we would often be told stories about applications not working with their supposedly "S3 compatible" one (from a mix of vendors).
Re:S3 is files, but not a filesystem by yencabulator on Sunday March 10, 2024 @02:39PM (#39661420)
Back in 2011 when I was working on making Ceph's RadosGW more S3-compatible, it was pretty common that AWS S3 behavior differed from their documentation too. I wrote a test suite to run against AWS and Ceph, just to figure out the differences. That lives on at https://github.com/ceph/s3-tests
Re:S3 is files, but not a filesystem by orf on Sunday March 10, 2024 @08:55PM (#39663823)
What differences in behaviour from the AWS docs did you find, out of interest?
Re:S3 is files, but not a filesystem by yencabulator on Sunday March 10, 2024 @09:10PM (#39663905)
What I can dig up today is that back in 2011, they documented that bucket names cannot look like IPv4 addresses and the character set was a-z0-9.-, but they failed to prevent 192.168.5.123 or _foo.

I recall there were more edge cases around HTTP headers, but they don't seem to have been recorded as test cases -- it's been too long for me to remember details, I may have simply ran out of time / real world interop got good enough to prioritize something else.

2011 state, search for fails_on_aws: https://github.com/tv42/s3-tests/blob/master/s3tests/functio...

Current state, I can't speak to the exact semantics of the annotations today, they could simply be annotating non-AWS features: https://github.com/ceph/s3-tests/blob/master/s3tests/functio...

Re:S3 is files, but not a filesystem by zaphar on Sunday March 10, 2024 @09:42AM (#39658967)
GCP storage buckets implement the S3 api. You can treat them like they were an s3 bucket. Something I do all the time.
Re:S3 is files, but not a filesystem by arcfour on Sunday March 10, 2024 @09:27AM (#39658869)
I've seen several S3-compatible APIs and there are open-source clients. If anything it's the de-facto standard.
Re:S3 is files, but not a filesystem by mechanicalpulse on Sunday March 10, 2024 @01:11PM (#39660737)
Isn't that a limitation imposed by the POSIX APIs, though, as a direct consequence of the interface's representation of hierarchical filesystems as trees? As you've illustrated, that necessitates walking the tree. Many tools, I suppose, walk the tree via a single thread, further serializing the process. In an admittedly haphazard test, I ran `find(1)` on ext4, xfs, and zfs filesystems and saw only one thread.

I imagine there's at least one POSIX-compatible file system out there that supports another, more performant method of dumping its internal metadata via some system call or another. But then we would no longer be comparing the S3 and POSIX APIs.

Re:S3 is files, but not a filesystem by jacobsimon on Monday March 11, 2024 @09:10AM (#39667774)
Interesting - isn't this just a matter of indexing/caching the file names, though? Surely S3 must store the files somewhere and index them. There's a Unix command called `locate` that does the same thing by maintaining a local database of keys and lets you search with prefixes.[1]

Anyway, I guess this is beyond the point of the original commenter above. I would disagree that listing files efficiently is the most useful part of S3. The main value prop is the fact that you can easily upload and download files from a distributed store. Most use cases involve uploading and downloading known files, not efficiently listing millions of files.

[1] https://jvns.ca/blog/2015/03/05/how-the-locate-command-works...

Re:S3 is files, but not a filesystem by aeyes on Sunday March 10, 2024 @08:56AM (#39658677)
And if for some reason you need a complete listing along with object sizes and other attributes you can get one every 24 hours with S3 inventory report.

That has always been good enough for me.

Re:S3 is files, but not a filesystem by harshaw on Monday March 11, 2024 @09:23AM (#39667919)
When you have billions of objects this is really the only way to go - and build workflows around inventory (including athena, spark, etc)
Re:S3 is files, but not a filesystem by tjoff on Sunday March 10, 2024 @08:57AM (#39658685)
Is listing really such a key feature that people use it as a database to find objects?

Have not used S3, but that is not how I imagined using it.

Re:S3 is files, but not a filesystem by orf on Sunday March 10, 2024 @09:07AM (#39658730)
Sure. It's kind of an index - limited to prefix-only searching, but useful.

Say you store uploads associated with a company and a user. You'd maybe naively store them as `[company-uuid]/[user-id].[timestamp]`.

If you need to list a given users (123) uploads after a given date, you'd list keys after `[company-uuid]/123.[date]`. If you need to list all users uploads, you'd list `[company-uuid]/123.`. If you need to get a set of all users who have photos, you'd list `[company-uuid]/` with a Delimiter set to `.`

The point is that it's flexible and with a bit of thought it allows you to "remove all a users uploads between two dates", "remove all a companies uploads" or "remove all a users uploads" with a single call. Or whatever specific stuff is important to your use-case, that might otherwise need a separate DB.

It's not perfect - you can't reverse the listing (i.e you can't get the latest photo for a given user by sorting descending for example), and needs some thought about your key structure.

Re:S3 is files, but not a filesystem by tjoff on Sunday March 10, 2024 @09:18AM (#39658817)
But surely you need to track that elsewhere anyway?

That some niche edge-case runs efficiently doesn't sound like a defining feature of S3. On the contrary many common operations map terrible to S3, so you kind of need the logic to be elsewhere.

Re:S3 is files, but not a filesystem by kbolino on Sunday March 10, 2024 @11:13AM (#39659790)
> But surely you need to track that elsewhere anyway?

Why? If the S3 structure and listing is sufficient, I don't need to store anything else anywhere else.

Many use cases may involve other requirements that S3 can't meet, such as being able to find the same object via different keys, or being able to search through the metadata fields. However, if the requirements match up with S3's structure, then additional services are unnecessary and keeping them in sync with S3 is more hassle than it's worth.

Re:S3 is files, but not a filesystem by tjoff on Sunday March 10, 2024 @11:36AM (#39660008)
I agree, but something as simple (in functionality) as that ought to be an edge-case. Not a defining feature of S3.
Re:S3 is files, but not a filesystem by orf on Sunday March 10, 2024 @12:01PM (#39660197)
It’s fundamental to how S3 works and its ability to scale, so it is a defining feature of S3.

If you think wider, a bucket itself is just a prefix.

Re:S3 is files, but not a filesystem by tjoff on Sunday March 10, 2024 @05:00PM (#39662486)
From amazons perspective, sure!

But that's not what we are discussing.

Re:S3 is files, but not a filesystem by dekhn on Sunday March 10, 2024 @11:56AM (#39660152)
it's a property of the system that I, as an architect, would seriously consider as part of my system's design. I've worked with many systems where iterating over items in order starting from a prefix is extremely cheap (sstables).
Re:S3 is files, but not a filesystem by orf on Sunday March 10, 2024 @10:10AM (#39659226)
My overall point can be summarised as this:

- Listing things is a very common operation to do.

- The POSIX api and the directory/file hierarchy it provides is a restrictive one.

- S3 does not suffer from this, you can recursively list and group keys into directories at “list time”.

- If you find yourself needing to list gigantic numbers of keys in one go, you can do better by only listing a subset. S3 isn’t a filesystem, you shouldn’t need to list 1k+ keys sequentially apart from during maintenance tasks.

- This is actually quite fast, compared to alternatives.

Whether or not you see a use case for this is sort of irrelevant: they exist. it’s what allows you to easily put data into s3 and flexibly group/scan it by specific attributes.

Re:S3 is files, but not a filesystem by tjoff on Sunday March 10, 2024 @10:29AM (#39659395)
Listing things is very common, so why would you outsource that to S3 when all your bookkeeping is elsewhere? It's not like you would ever rely on the POSIX API for that anyway, even for when your files actually are on a POSIX filesystem.

For sure, for maintenance tasks etc. it sounds quite useful. And good hygiene with prefixes sounds like a sane idea. But listing being a critical part of what "makes S3 useful"? That seems like an huge stretch that your points don't seem to address.

Re:S3 is files, but not a filesystem by orf on Sunday March 10, 2024 @02:10PM (#39661189)
> It's not like you would ever rely on the POSIX API for that anyway, even for when your files actually are on a POSIX filesystem.

Because there is no POSIX api for this. Depending on your requirements and query patterns, you may not need a completely separate database that you need to keep in sync.

Re:S3 is files, but not a filesystem by IanCal on Monday March 11, 2024 @10:12PM (#39675483)
You may not need other bookkeeping. The prefix listing properties can be enough, removing the need to have two distinct systems kept in sync.
Re:S3 is files, but not a filesystem by belter on Sunday March 10, 2024 @10:06AM (#39659187)
No. The standard practice is to use a DynamoDB table as the index for your objects in S3.

This article misunderstood S3 and could as well have the title: "An Airplane is not a Car" :-)

Re:S3 is files, but not a filesystem by macintux on Sunday March 10, 2024 @11:35AM (#39660006)
I don't know that you can characterize that as a "standard practice".

Maybe it's widespread, but I've not encountered it.

Re:S3 is files, but not a filesystem by belter on Sunday March 10, 2024 @12:08PM (#39660245)
Here is the architecture of Amazon Drive and the storage of metadata.

"AWS re:Invent 2014 | (ARC309) Building and Scaling Amazon Cloud Drive to Millions of Users" - https://youtu.be/R2pKtmhyNoA

And you can see the use here at correct time: https://youtu.be/R2pKtmhyNoA?t=546

Re:S3 is files, but not a filesystem by ianburrell on Sunday March 10, 2024 @01:38PM (#39660921)
That article is old. DynamoDB was used because of the old, weak consistency model of S3. Writes were atomic, but lists could return old results so needed consistent list of objects.

But in 2020, S3 changed to strong consistency model. There is no need to use DynamoDB now.

Re:S3 is files, but not a filesystem by belter on Sunday March 10, 2024 @08:45PM (#39663758)
The problem was not the eventual consistency model, was the speed of the object list.

"...Finding objects based on other attributes, however, requires doing a linear search using the LIST operation. Because each listing can return at most 1000 keys, it may require many requests before finding the object. Because of these additional requests, implementing attribute-based queries in S3 alone can be challenging..."

Re:S3 is files, but not a filesystem by orf on Sunday March 10, 2024 @08:52PM (#39663812)
It was both actually, but more for the listing issue. Netflix built a lot of tooling around this.

But yeah: things like filtering on tags or created at dates requires another approach.

Re:S3 is files, but not a filesystem by mr_toad on Monday March 11, 2024 @12:53AM (#39665016)
Hive stores metadata in a relational database. So does Snowflake.
Re:S3 is files, but not a filesystem by fijiaarone on Sunday March 10, 2024 @03:33PM (#39661891)
So in reality S3 takes about 2 seconds to retrieve a single file, under ideal conditions. 1 second round trip for the request to DynamoDB to get the object key of the file and 1 second round trip to S3 to get the file contents (assuming no CPU cost on the search because you’re getting the key by ID from the DynamoDB in a flat single table store. And that the file has no network IO because it is a trivial number of bytes, so the HTTP header overwhelmed the content.)

I know what you’re thinking — 2 seconds, that’s faster than I can type the 300 character file key with its pseudo prefixes)!

Ah, but what if you wanted to get 2 files from S3?

Re:S3 is files, but not a filesystem by orf on Sunday March 10, 2024 @08:53PM (#39663817)
2 seconds is a nuts response time, but I guess it depends entirely on your file size. TTFB is usually 50ms.
Re:S3 is files, but not a filesystem by deathanatos on Sunday March 10, 2024 @10:18PM (#39664336)
… S3's response time is nowhere near 2 seconds. (Or even 1 second.) Like a sibling poster says, 50ms is a much more realistic ballpark for TTFB.
Re:S3 is files, but not a filesystem by calpaterson on Sunday March 10, 2024 @09:14AM (#39658779)
I have to say that I'm not hugely convinced. I don't really think that being able to pull out the keys before or after a prefix is particularly impressive. That is the basis for database indices going back to the 1970s after all.

Perhaps the use-cases you're talking about are very different from mine. That's possible of course.

But for me, often the slow speed of listing the bucket gets in the way. Your bucket doesn't have to get very big before listing the keys takes longer than reading them. I seem to remember that listing operations ran at sub-1mbps, but admittedly I don't have a big bucket handy right now to test that.

Re:S3 is files, but not a filesystem by orf on Sunday March 10, 2024 @09:59AM (#39659121)
It depends on a few factors. The list objects call hides deleted and noncurrent versions, but it has to skip over them. Grouping prefixes also takes time, if they contain a lot of noncurrent or deleted keys.

A pathological case would be a prefix with 100 million deleted keys, and 1 actual key at the end. Listing the parent prefix takes a long time in this case - I’ve seen it take several minutes.

If your bucket is pretty “normal” and doesn’t have this, or isn’t versioned, then you can do 4-5 thousand list requests a second, at any given key/prefix, in constant time. Or or you can explicitly list object versions (and not skip deleted keys) also in constant time.

It all depends on your data: if you need to list all objects then yeah it’s gonna be slow because you need to paginate through all the objects. But the point is that you don’t have to do that if you don’t want to, unlike a traditional filesystem with a directory hierarchy.

And this enables parallelisation: why list everything sequentially, when you can group the prefixes by some character (i.e “-“), then process each of those prefixes in parallel.

The world is your oyster.

Re:S3 is files, but not a filesystem by cuno on Sunday March 10, 2024 @10:22AM (#39659324)
We and our customers use S3 as a POSIX filesystem, and we generally find it faster than a local filesystem for many benchmarks. For listing directories we find it faster than Lustre (a real high performance filesystem). Our approach is to first try listing directories with a single ListObjectV2 (which on AWS S3 is in lexicographic order) and if it hasn't made much progress, we start listing with parallel ListObjectV2. Once you start parallelising the ListObjectV2 (rather than sequentially "continuing") you get massive speedups.
Re:S3 is files, but not a filesystem by supriyo-biswas on Sunday March 10, 2024 @11:13AM (#39659801)
> Once you start parallelising the ListObjectV2 (rather than sequentially "continuing")

How are you "parallelizing" the ListObjectsV2? The continuation token can be only fed in once the previous ListObjectsV2 response has completed, unless you know the name or structure of keys ahead of time, in which listing objects isn't necessary.

Re:S3 is files, but not a filesystem by johnmaguire on Sunday March 10, 2024 @11:21AM (#39659874)
You're right that it won't work for all use cases, but starting two threads with prefixes A and M, for example, is one way you might achieve this.
Re:S3 is files, but not a filesystem by cuno on Sunday March 10, 2024 @11:20AM (#39659869)
For example, you can do separate parallel ListObjectV2 for files starting a-f and g-k, etc.. covering the whole key space. You can parallelize recursively based on what is found in the first 1000 entries so that it matches the statistics of the keys. Yes there may be pathological cases, but in practice we find this works very well.
Re:S3 is files, but not a filesystem by crabbone on Sunday March 10, 2024 @10:51AM (#39659600)
> find it faster than a local filesystem for many benchmarks.

What did you measure? How did you compare? This claim seems very contrary to my experience and understanding of how things work...

Let me refine the question: did you measure metadata or data operations? What kind of storage medium is used by the filesystem you use? How much memory (and subsequently the filesystem cache) does your system have?

----

The thing is: you should expect, in the best case, something like 5 ms latency on network calls over the Internet in an ideal case. Within the datacenter, maybe you can achieve sub-ms latency, but that's hard. AWS within region but different zones tends to be around 1 ms latency.

This is while NVMe latency, even on consumer products, is 10-20 micro seconds. I.e. we are talking about roughly 100 times faster than anything going through the network can offer.

Re:S3 is files, but not a filesystem by cuno on Sunday March 10, 2024 @11:00AM (#39659684)
For AWS, we're comparing against filesystems in the datacenter - so EBS, EFS and FSx Lustre. Compared to these, you can see in the graphs where S3 is much faster for workloads with big files and small files: https://cuno.io/technology/

and in even more detail of different types of EBS/EFS/FSx Lustre here: https://cuno.io/blog/making-the-right-choice-comparing-the-c...

Re:S3 is files, but not a filesystem by hnlmorg on Sunday March 10, 2024 @11:22AM (#39659882)
EFS is ridiculously slow though. Almost to the point where I fail to see how it’s actually useful for any of the traditional use cases for NFS.
Re:S3 is files, but not a filesystem by dekhn on Sunday March 10, 2024 @11:54AM (#39660132)
if you turn all the EFS performance knobs up (at a high cost), it's quite fast.
Re:S3 is files, but not a filesystem by hnlmorg on Sunday March 10, 2024 @12:24PM (#39660375)
Faster, sure. But I wouldn’t got so far as to say it is fast
Re:S3 is files, but not a filesystem by harshaw on Monday March 11, 2024 @09:09AM (#39667764)
Have you tried it recently? Because we've made it a lot faster over the years.
Re:S3 is files, but not a filesystem by hnlmorg on Monday March 11, 2024 @01:25PM (#39670969)
More recently and for more use cases and varied workflows than most people. But that’s as much as I can say without getting people to sign an NDA.

Our AWS spend is high enough to warrant a very close working relationship with AWS so this is something we have worked with you guys on already.

Re:S3 is files, but not a filesystem by geertj on Monday March 11, 2024 @11:31AM (#39669454)
> EFS is ridiculously slow though. Almost to the point where I fail to see how it’s actually useful for any of the traditional use cases for NFS.

Would you care to elaborate on your experience or use case a bit more? We've made a lot of improvements over the last few years (and are actively working on more), and we have many happy customers. I'd be happy to give a perspective of how well your use case would work with EFS.

Source: PMT turned engineer on EFS, with the team for over 6 years

Re:S3 is files, but not a filesystem by hnlmorg on Monday March 11, 2024 @01:34PM (#39671079)
Unfortunately I can’t say too much publicly on HN. But one of the big shortcomings is dealing with hundreds of files. It doesn’t even matter if those are big or small files (I’ve had experience with both).

Services like DataSync show that the underlying infra can be performant. But it feels almost impossible to replicate that on EFS via standard POSIX APIs. And unfortunately one of our use cases depend upon that.

If feels, to me at least, like EFS isn’t where AWSs priorities lie. At least if you compare EFS to FSx Lustre and recent developments to S3. Both of which has been the direction our AWS SAs have pushed us.

Re:S3 is files, but not a filesystem by wenc on Sunday March 10, 2024 @01:35PM (#39660908)
S3 is really high latency though. I store parquet files on S3 and querying them through DuckDB is much slower than file system because random access patterns. I can see S3 being decent if it’s bulk access but definitely not for random access.

This is why there’s a new S3 Express offering that is low latency (but costs more).

Re:S3 is files, but not a filesystem by crabbone on Sunday March 10, 2024 @02:03PM (#39661129)
The tests are very weird...

Normally, from someone working in the storage, you'd expect tests to be in IOPS, and the goto tool for reproducible tests is FIO. I mean, of course "reproducibility" is a very broad subject, but people are so used to this tool that they develop certain intuition and interpretation for it / its results.

On the other hand, seeing throughput figures is kinda... it tells you very little about how the system performs. Just to give you some reasons: a system can be configured to do compression or deduplication on client / server, and this will significantly impact your throughput, depending on what do you actually measure: the amount of useful information presented to the user or the amount of information transferred. Also throughput at the expense of higher latency may or may not be a good thing... Really, if you ask anyone who ever worked on a storage product about how they could crank up throughput numbers, they'd tell you: "write bigger blocks asynchronously". This is the basic recipe, if that's what you want. Whether this makes a good all around system or not... I'd say, probably not.

Of course, there are many other concerns. Data consistency is a big one, and this is a typical tradeoff when it comes to choosing between object store and a filesystem, since filesystem offers more data consistency guarantees, whereas object store can do certain things faster, while breaking them.

BTW, I don't think most readers would understand Lustre and similar to be the "local filesystem", since it operates over network and network performance will have a significant impact, of course, it will also put it in the same ballpark as other networked systems.

I'd also say that Ceph is kinda missing from this benchmark... Again, if we are talking about filesystem on top of object store, it's the prime example...

Re:S3 is files, but not a filesystem by cuno on Sunday March 10, 2024 @05:12PM (#39662573)
IOPS is a really lazy benchmark that we believe can greatly diverge from most real life workloads, except for truly random I/O in applications such as databases. For example, in Machine Learning, training usually consists of taking large datasets (sometimes many PBs in scale), randomly shuffling them each Epoch, and feeding them into the engine as fast as possible. Because of this, we see storage vendors for ML workloads concentrate on IOPS numbers. The GPUs however only really care about throughput. Indeed, we find a great many applications only really care about the throughput, and IOPS is only relevant if it helps to accomplish that throughput. For ML, we realised that the shuffling isn't actually random - there's no real reason for it to be random versus pseudo-random. And if its pseudo-random then it is predictable, and if its predictable then we can exploit that to great effect - yielding a 60x boost in throughput on S3, beating out a bunch of other solutions. S3 is not going to do great for truly random I/O, however, we find that most scientific, media and finance workloads are actually deterministic or semi-deterministic, and this is where cunoFS, by peering inside each process, can better predict intra-file and inter-file access patterns, so that we can hide the latencies present in S3. At the end of the day, the right benchmark is the one that reflects real world usage of applications, but that's a lot of effort to document one by one.

I agree that things like dedupe and compression can affect things, so in our large file benchmarks each file is actually random. The small file benchmarks aren't affected by "write bigger blocks" because there's nothing bigger than the file itself. Yes, data consistency can be an issue, and we've had to do all sorts of things to ensure POSIX consistency guarantees beyond what S3 (or compatible) can provide. These come with restrictions (such as on concurrent writes to the same file on multiple nodes), but so does NFS. In practice, we introduced a cunoFS Fusion mode that relies on a traditional high-IOPS filesystem for such workloads and consistency (automatically migrating data to that tier), and high throughput object for other workloads that don't need it.

Re:S3 is files, but not a filesystem by rfoo on Monday March 11, 2024 @05:59AM (#39666291)
> And if its pseudo-random then it is predictable, and if its predictable then we can exploit that to great effect

This is an interesting hack. However, an IOP is an IOP, no matter how good you predicted it and prefetch it so that you hide the latency it's going to be translated to a GetObject.

I think what you really exploited here is that even though S3 is built on HDDs (and have very low IOPS per TiB) their scale is so large that even if you milk 1M+ IOPS out of it AWS still doesn't care and is happy to serve you. But if my back-of-envelope calculation is correct this isn't going to work well if everyone starts to do it.

How do you get around S3's 5.5k GET per second per prefix limit? If I only have ~200 20GiB files can you still get decent IOPS out of it?

and...

> IOPS is a really lazy benchmark that we believe can greatly diverge from most real life workloads

No, it's not. I have a workload training a DL model on time series data which demands 600k 8KiB IOPS per compute instance. None of the thing I tested work well. Had to build a custom one with bare metal NVMe-s.

Re:S3 is files, but not a filesystem by cuno on Wednesday March 13, 2024 @07:56AM (#39690384)
Sorry for the late response - I didn't see your comment until now.

Our aim is to unleash all the potential that S3/Object has to offer for file system workloads. Yes, the scale of AWS S3 helps, as does erasure coding (which enhances flexibility for better load balancing of reads).

Is it suitable for every possible workload? No, which is why we have a mode called cunoFS Fusion where we let people combine a regular high-performance filesystem for IOPS, and Object for throughput, with data automatically migrated between the two according to workload behaviour. What we find is that most data/workloads need high throughput rather than high IOPS, and this tends to be the bulk of data. So rather than paying for PBs of ultra-high IOPS storage, they only need to pay for TBs of it instead. Your particular workload might well need high IOPS, but a great many workloads do not. We do have organisations doing large scale workloads on time-series (market) data using cunoFS with S3 for performance reasons.

Re:S3 is files, but not a filesystem by fijiaarone on Sunday March 10, 2024 @03:20PM (#39661772)
If you think s3 is fast, you should try FTP. It’s at least a hundred times faster. And combined with rsync, dozens of times more reliable.
Re:S3 is files, but not a filesystem by orf on Sunday March 10, 2024 @08:48PM (#39663787)
Neither of those are true though? Not sure if this is sarcastic or not, if so make it more clear in the future
Re:S3 is files, but not a filesystem by YZF on Monday March 11, 2024 @01:46AM (#39665226)
It can't be a POSIX filesystem if it doesn't meet POSIX filesystem guarantees. I worked on an S3 compatible object store in a large storage company and we also had distributed filesystem products. Those are completely different animals due to the different semantics and requirements. We've also built compliant filesystems over object store and the other way around. Certain operations like, write-append, are tricky to simulate over object stores (S3 didn't use to support append, I haven't really stayed up to date, does it now?). At least when I worked on this it wasn't possible to simulate POSIX semantics over S3 at all without needing to add additional object store primitives.
Re:S3 is files, but not a filesystem by hayd on Sunday March 10, 2024 @09:43AM (#39658980)
You can set up cloud watch events to trigger a lambda function to store meta data about the s3 file in a regular database. That way you can index it how you expect to list.

Very effective for our use case.

Re:S3 is files, but not a filesystem by foldr on Sunday March 10, 2024 @10:22AM (#39659330)
>What makes it useful is listing.

I think 99% of S3 usage just consists of retrieving objects with known keys. It seems odd to me to consider prefix listing as a key feature.

Re:S3 is files, but not a filesystem by bostik on Sunday March 10, 2024 @10:31AM (#39659414)
When you embed the relevant (not necessarily that of object creation) timestamp as a prefix, it sure becomes one. Whether that prefix is part of the "path" (object/path/prefix/with/<4-digit year/)" or directly part of the basename (object/path/prefix/to/app-specific/files/<4-digit year>-<2-digit month>-....), being able to limit the search space server-side becomes incredibly useful.

You can try it yourself: list objects in a bucket prefix with lots of files, and measure the time it takes to list all of them vs. the time it takes to list only a subset of them that share a common prefix.

Re:S3 is files, but not a filesystem by gamache on Sunday March 10, 2024 @11:46AM (#39660080)
> ...listing any given prefix is essentially constant time: I can take any given string, in a bucket with 100 billion objects, and say “give me the next 1000 keys alphabetically that come after this random string”.

I'm not sure we agree on the definition of "constant time" here. Just because you get 1000 keys in one network call doesn't imply anything about the complexity of the backend!

Re:S3 is files, but not a filesystem by orf on Sunday March 10, 2024 @11:56AM (#39660151)
Constant time irregardless of the number of objects in the bucket and irregardless of the initial starting position of your list request.
Re:S3 is files, but not a filesystem by hobobaggins on Sunday March 10, 2024 @12:16PM (#39660317)
The technical implementation is indeed impressive that it operates more-or-less within constant time, but probably very few use cases actually fit that narrow window, so this technical strength is moot when it comes to actual usage.

Since each request is dependent upon the position received in the last request, 1000 arbitrary keys on your 3rd or 1000th attempt doesn't really help unless you found your needle in the haystack in that request (and in that case the rest of that 1000 key listing was wasted.)

Re:S3 is files, but not a filesystem by orf on Sunday March 10, 2024 @12:48PM (#39660567)
You’re assuming you’re paginating through all objects from start to finish.

A request to list objects under “foo/“ is a request to list all objects starting with “foo/“, which is constant time irregardless of the number of keys before. Same applies for “foo/bar-“, or any other list request for any given prefix. There are no directories on s3.

Re:S3 is files, but not a filesystem by nh2 on Sunday March 10, 2024 @12:33PM (#39660445)
The key difference between lexicographically keyed flat hierarchies, and directory-nested filesystem hierarchies, becomes clear based on this example:

    dir1/a/000000
    dir1/a/...
    dir1/a/999999
    dir1/b
On a proper hierarchical file file system with directories as tree interior nodes, `ls dir1/` needs to traverse and return only 2 entries ("a" and "b").

A flat string-indexed KV store that only supports lexicographic order, without special handling of delimters, needs to traverse 1 million dirents ("a/00000" throuh "a/999999") before arriving at "b".

Thus, simple flat hierarchies are much slower at listing the contents of a single dir: O(all recursive children), vs. O(immediate children) on a "proper" filesystem.

Lexicographic strings cannot model multi-level tree structures with the same complexities; this may give it the reputation of "listing files is slow".

UNLESS you tell the listing algorithm what the delimter character is (e.g. `/`). Then a lexicographical prefix tree can efficiently skip over all subtrees at the next `/`.

Amazon S3 supports that, with the docs explicitly mentioning "skipping over and summarizing the (possibly millions of) keys nested at deeper levels" in the `CommonPrefixes` field: https://docs.aws.amazon.com/AmazonS3/latest/userguide/using-...

I have not tested whether Amazon's implemented actually saves the traversal (or whether it traverses and just returns less results), but I'd hope so.

Re:S3 is files, but not a filesystem by nh2 on Sunday March 10, 2024 @01:46PM (#39660991)
For completeness: The orignal post says:

    S3 has no rename or move operation.
    Renaming is CopyObject and then DeleteObject.
    CopyObject takes linear time to the size of the file(s).
    This comes up fairly often when someone has written a lot of files
    to the wrong place - moving the files back is very slow.
This is right:

In a normal file system, renaming a directory is fast O(1), in S3 it's slow O(all recursive children).

And Amazon S3 has not added a delimiter-based function to reduce its complexity, even though that would be easily possible in a lexicographic prefix tree (re-rooting the subtree).

So here the original post has indeed found a case where S3 is much slower than a normal file system.