I’m more curious if there is an equivalent to S3 for on-prem. I bet not due to reliability constraints, but something like this would be nice. I can’t say there would be much benefit to using AWS Outposts for genome sequencing without an S3 local cache.
If you are asking if there are stand-alone S3 API offerings, you might want to look at MinIO. Reliability mostly requires extra hardware and configuration - https://docs.min.io/docs/minio-erasure-code-quickstart-guide...
> 380 TB S3 tier is priced at $0.1000/GB-mo ($38,912.00/month)
But it's also extremely expensive. $38K per month for 380 TB of storage. Even if we assume 3X redundancy, you can get 500TB of raw capacity (12-16TB HDDs) for ~ $25k. After 3 months of usage, you'd have already covered that initial expense. I know there are other benefits to having managed services, but again, cost is definitely prohibitive for many use-cases.
I use minio locally to support S3-workloads on our local cluster. We can then migrate workloads to AWS/S3 proper with minimal changes, if needed.
The ENCODE Project data were a hundred terabytes or so at the point I migrated it into S3 (it's now over a petabyte.) We were on a fast academic network with a 100GB interconnect so could just `aws s3 sync` rather than worry about using AWS Snowballs or anything.
With Outpost this becomes feasible. This "cloud in a box" idea has been around for a long time, but I guess Amazon has enough brainshare for the tooling that it may make sense for companies to have this for data-intensive compute and keep the same skillset requirements as the rest of their fully-clouded operations.