News for hackers. Stuff that matters.
Re:AWS Outposts by r00fus on Tuesday November 30, 2021 @05:56PM (#29398506)
Yes - take for example gene sequencing - that requires a huge amount of data shuffling and going cloud for that is infeasible simply for the data transfer costs.

With Outpost this becomes feasible. This "cloud in a box" idea has been around for a long time, but I guess Amazon has enough brainshare for the tooling that it may make sense for companies to have this for data-intensive compute and keep the same skillset requirements as the rest of their fully-clouded operations.

Re:AWS Outposts by mbreese on Tuesday November 30, 2021 @09:59PM (#29400159)
I do genome sequencing and this is very true. You are either all-in on cloud or do the processing completely in house. The data egress fees are more than enough to offset any benefit to a hybrid approach.

I’m more curious if there is an equivalent to S3 for on-prem. I bet not due to reliability constraints, but something like this would be nice. I can’t say there would be much benefit to using AWS Outposts for genome sequencing without an S3 local cache.

Re:AWS Outposts by matt_heimer on Tuesday November 30, 2021 @10:55PM (#29400487)
If you are asking if Outputs includes S3, yes, https://aws.amazon.com/s3/outposts/

If you are asking if there are stand-alone S3 API offerings, you might want to look at MinIO. Reliability mostly requires extra hardware and configuration - https://docs.min.io/docs/minio-erasure-code-quickstart-guide...

Re:AWS Outposts by mbreese on Wednesday December 01, 2021 @12:16PM (#29406337)
I didn't know they also had S3 for outposts... that's interesting. That also requires an Outposts rack as opposed to the servers the original article was talking about.

> 380 TB S3 tier is priced at $0.1000/GB-mo ($38,912.00/month)

But it's also extremely expensive. $38K per month for 380 TB of storage. Even if we assume 3X redundancy, you can get 500TB of raw capacity (12-16TB HDDs) for ~ $25k. After 3 months of usage, you'd have already covered that initial expense. I know there are other benefits to having managed services, but again, cost is definitely prohibitive for many use-cases.

I use minio locally to support S3-workloads on our local cluster. We can then migrate workloads to AWS/S3 proper with minimal changes, if needed.

Re:AWS Outposts by r00fus on Wednesday December 01, 2021 @04:29PM (#29409358)
How do you even get the petabytes of data into S3 without incurring massive charges? Previous estimations of getting the initial dataset into cloud was.. prohibitive (note this was 3-4 years ago).
Re:AWS Outposts by laurencerowe on Thursday December 02, 2021 @02:52PM (#29421525)
What charges in particular are you referring to? The cloud services only charge data egress fees. Data ingress is free.

The ENCODE Project data were a hundred terabytes or so at the point I migrated it into S3 (it's now over a petabyte.) We were on a fast academic network with a 100GB interconnect so could just `aws s3 sync` rather than worry about using AWS Snowballs or anything.

Re:AWS Outposts by nicornk on Thursday December 02, 2021 @02:56PM (#29421606)
Uploading data into AWS is free.