Every story and comment from the Hacker News API, in a layout borrowed from 2001.
A comment on S3 is files, but not a filesystem. Parent comment
There's no "recursive" nature to S3 buckets. "Listing a directory" is simply listing keys by a prefix.
So list by the upper-most prefix that you want. If you have 1,000,000 files, it will take 1,000 API calls to list everything.
If each call takes 1s (I have no idea what your latency to the S3 bucket region is), then it will indeed take 15 min.
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObje...
There are no comments on this comment yet.
There's no "recursive" nature to S3 buckets. "Listing a directory" is simply listing keys by a prefix.
So list by the upper-most prefix that you want. If you have 1,000,000 files, it will take 1,000 API calls to list everything.
If each call takes 1s (I have no idea what your latency to the S3 bucket region is), then it will indeed take 15 min.
https://docs.aws.amazon.com/AmazonS3/latest/API/API_ListObje...