News for hackers. Stuff that matters.
Re:Command-line tools can be faster than a Hadoop cluster (2... by Johnny555 on Thursday January 30, 2020 @03:41AM (#22189872)
We didn't find that managed elasticsearch saved us any money over running our own ES cluster, but the best part of the cloud for my company is the scalability -- our peak monthly load is 10X more than our light weekend load. So to accommodate that scalability, instead of buying and maintaining 1000 servers to handle our light base load, we'd need to own 10,000 servers that would only be less than 30% utilized on average. And for redundancy, we'd need to spread those servers across multiple data centers, as well as manage a 2 petabyte storage system (also replicated across data centers) to replace our S3 usage.

And we'd need to have staff to manage the physical servers, run our network infrastructure, etc.

We've been through the numbers many times and the cloud always wins, even when compared to running our own servers for base load and have some hybrid cloud that lets us expand into AWS for peak loads. (which saves some dollars in servers, but that doesn't make up for the added complexity).

And nearly for "free", we have a cold standby site that has a replica of our hot data (a small portion of our full dataset), so if there was a full region outage, we could be back up in running in 30 minutes with reduced functionality.

Re:Command-line tools can be faster than a Hadoop cluster (2... by La1n on Thursday January 30, 2020 @05:35AM (#22190393)
As someone who doesn't know a lot about this stuff, is there a way to have the base load internal and use a cloud provider for elastic load?
Re:Command-line tools can be faster than a Hadoop cluster (2... by krab on Thursday January 30, 2020 @06:48AM (#22190707)
Yes, with a caveat which is latency. For example it's hard to split a typical webapp so the application servers are far from the database. The latency kills performance.

On the other hand, for stuff like CI jobs, batch processing or anything else which doesn't depend on tight synchronization with the other location, you can mix and match.