News for hackers. Stuff that matters.
Re:A terrible, horrible, no-good, very bad day at Slack by gchamonlive on Tuesday July 07, 2020 @05:51AM (#23757394)
You can't assume that, these costs, specifically the server less stack at AWS, scale up quite opaquely. You can put billing alerts in place, but once you tie your infrastructure to one specific serverless vendor, even if you identify a harmful cost scale, you can't easily mitigate it.

I am not saying serverless is expensive, all I am advocating is extensive planning and preparing before adopting any particular serverless solution. Once you give the green light to adopt a specific cloud solution, you tie yourself to that cloud, and that can turn out to be a bad idea in the long run.

Lambda is a great tool when used right. It can take big workloads without costing too much. But in itself it does very little. It still needs to intergrate with something to be triggered, and if you go with AWS API Gateway that can cost you a pretty penny. Load balancer also incurs costs that are difficult to predict and is not so flexible as other load balancing tools, so sometimes you might still need to provision you own service discovery and load distribution layers, just like they had to do at slack.

Serverless is nice, solves a lot of issues and give smaller teams a shot they otherwise wouldn't have, having to manage everything by themselves. But cloud costs can be opaque, cloud implementation can be very complex, cloud solutions can be too rigid sometimes and tying your product to one vendor can be detrimental in the long run.

Re:A terrible, horrible, no-good, very bad day at Slack by jjoonathan on Tuesday July 07, 2020 @07:33AM (#23758083)
Not just costs! AWS services are opaque performance minefields, too.

If you need reliable low latency -- which is probably core to the slack experience, or any GUI in a competitive space, really -- lambda is not a good option. They just don't share your priorities. AWS support will happily waste a lot of your time chasing Just One More Trick to mitigate the problem, though.

Re:A terrible, horrible, no-good, very bad day at Slack by ryan_lane on Tuesday July 07, 2020 @09:09AM (#23758813)
Lock-in is one of the worst reasons to avoid serverless. If you're really putting in the effort to avoid lock-in, you're wasting engineering time that could be better spent on your product. AWS (or GCP, or whatever cloud provider you choose) has a number of really great products that ultimately save you time and help you get to market faster and better.

It doesn't matter that you're locked-in, with slightly higher than wanted costs when you've failed due to poor priorities.

Become successful, then worry about removing your lock-in if you actually need it (and you probably won't).

Re:A terrible, horrible, no-good, very bad day at Slack by yjftsjthsd-h on Tuesday July 07, 2020 @10:01AM (#23759361)
I'd disagree pretty strongly. You may evaluate the tradeoffs and decide to go all-in on a single cloud (and you should do that analysis!), but it's far from a given. You're not wasting engineering time if the costs get out of hand and bankrupt you. Managed services may accelerate your time to market, but they do so at the expense of lock-in and your bill.

Context: I work at a startup that benefits enormously by avoiding AWS/GCP (for most cases) and renting cheap dedicated servers. It is context-dependent; our exact business doesn't benefit much from managed services and really needs big servers.

Re:A terrible, horrible, no-good, very bad day at Slack by ryan_lane on Tuesday July 07, 2020 @10:32AM (#23759657)
Just because you're using managed services doesn't mean it's going to be considerably more expensive, when you consider labor costs, and general reliability.

Your service will likely be more reliable if you use DynamoDB or AuroraDB. Your service will probably be more reliable if you build it in a way that assumes nodes will die at any point, will automatically come back in, and can scale up/down. It'll likely be more reliable if you use SQS rather than your own message bus (and let's be honest, it'll probably be cheaper too).

Yes, you should always evaluate the costs, but reliability and the amount of time you're going to spend maintaining something is something that somehow always gets left out of these evaluations.

Re:A terrible, horrible, no-good, very bad day at Slack by yjftsjthsd-h on Tuesday July 07, 2020 @11:10AM (#23760045)
(Prices based on https://calculator.aws/#/createCalculator and https://www.hetzner.com/dedicated-rootserver?country=us)

An AuroraDB db.r5.xlarge with 10TB of storage, reserved instances 1Y term but no up-front, costs 1,301.40 USD per month.

Take a Hetzner AX161 with 4x3.84 TB SATA SSD, using RAIDZ for 11.52TB usable storage (and 4 times the RAM), at €297.00 per month... so 335.88 USD per month.

That's a difference of 965 $/mo = 11,580 $/yr. If you have 10 of these, they'll pay for a full time sysadmin. Now, that's leaving out a lot of details (bandwith costs and application hosting come to mind), and assumes truly massive databases. On the other hand, as that sysadmin, I promise our databases don't take anything like my full attention, and you really should have some sort of sysadmin/ops team anyways (please do not make devs run your AWS infrastructure; it will end in tears for everyone). Every time this argument comes up, people do mention reliability and time spent on maintenance, but... it's really not bad. Hardware doesn't actually fail that much, postgres isn't that complicated to configure, OS patches aren't that hard to apply. Your mileage will vary, but sometimes it's just not worth using AWS. (And sometimes, it really is; if we didn't need to run oversized databases, I'd push us to use AWS in a heartbeat)

Re:A terrible, horrible, no-good, very bad day at Slack by EB66 on Tuesday July 07, 2020 @12:35PM (#23760898)
100% this. Cloud vendor lock-in is absolutely a relevant factor for consideration -- especially when the non-cloud alternatives are ~10x cheaper.

At my company we made the decision to stick primarily with managed dedicated servers over AWS in our very early days. Now we're a decent size a few years later (25 employees) and the cost savings we're realizing are tremendous. We did the math and found that if we had gone with AWS in the early days then we would now conservatively be paying an extra $165,000 on our hosting bill annually.

We still use AWS for some specialized services (e.g., Lex) but the bulk of our stack runs on gear that we now colocate for a fraction of the cost.

Re:A terrible, horrible, no-good, very bad day at Slack by bscanlan on Tuesday July 07, 2020 @04:02PM (#23763369)
You are forgetting to price in some minor features that Aurora provides: - Aurora's storage is spread across three availability zones. - Backups. - Automatic failover. - No need to configure anything, it just works.

If your time is free, and you don't actually need anything resembling high availability for the data in the database, then that's a good price comparison. I'm not arguing that managed databases makes sense for everybody, but if you're doing a price comparison then at least factor in multi-site redundancy for the data?

Re:A terrible, horrible, no-good, very bad day at Slack by yjftsjthsd-h on Tuesday July 07, 2020 @05:01PM (#23764026)
> You are forgetting to price in some minor features that Aurora provides

That's true and fair, although in both directions; skimming the docs it looks like aurora prices include 2 replicas? But backups aren't free (to store), bandwidth isn't free, and iops aren't free. Also, my difficulty in figuring out a fair pricing comparison highlights another point: a dedicated server has a fixed price. Other than more servers for more instances/replicas, you're never going to pay more, and even then it's a simple "adding another replica will increase our costs to X*(N+1) per month", not a "scaling out will add X to our costs, but if we use more I/O than expected we'll add Y to our costs, and exporting data will cost Z in bandwidth".

Re:A terrible, horrible, no-good, very bad day at Slack by cardine on Tuesday July 07, 2020 @03:00PM (#23762805)
This matches my experiences entirely. By running our own hardware (mix of Hetzner and colocation) we go from 20% of our revenue spent on server costs to less than 5%. Well worth any potential headaches, which are almost always overstated.
Re:A terrible, horrible, no-good, very bad day at Slack by lewis1028282 on Tuesday July 07, 2020 @01:45PM (#23761895)
Yeah but why pay someone to look after databases when I could get them do something else that'll have way more value. Using managed services just reduces how much money I have to spend upfront.
Re:A terrible, horrible, no-good, very bad day at Slack by yjftsjthsd-h on Tuesday July 07, 2020 @04:57PM (#23763978)
It depends™ on your (human and computer) workloads, but in my experience you need ops people to keep everything working smoothly, and once you have them it's easy to have them do this stuff without taking away from other areas. YMMV.
Re:A terrible, horrible, no-good, very bad day at Slack by FuckButtons on Tuesday July 07, 2020 @10:33AM (#23759671)
Getting to market faster is fine for an early startup, less so for a public company which needs to justify costs to its shareholders.
Re:A terrible, horrible, no-good, very bad day at Slack by gchamonlive on Tuesday July 07, 2020 @01:23PM (#23761566)
Not in the scenario I described. You implemented a solution using lambda and api gateway. Hundreds of microservices later you may realise you could save considerable money going unmanaged. By that time your lock in is costing you an unhealthy amount of money and there is no easy roadmap to mitigate it.

Again, everything must be planned beforehand. Savings could also be marginal, but it could also be significant. In big enterprises, where billing is north of a couple of millions of dollars, every percentile you can save is justifiable.

Re:A terrible, horrible, no-good, very bad day at Slack by deleuze on Tuesday July 07, 2020 @10:45AM (#23759787)
Not every company is a startup dude.