For 99% of projects you're never going to hit the point a single server (or group of servers if you truly need redundancy for some level of uptime) can't handle the load. For the other 1% that end up needing that scale I have a hard time accepting it's actually better to start building for massive scale day 1 instead of day 1000.

This is such a narrow perspective. It’s not always about load, most applications can benefit from some level of redundancy and using AWS etc doesn’t equal a massive scaling operation forced upon you. In fact, the other options would. I'll give you some common examples.

You have a script that needs to run, without fail, at a certain time every day.

You have an endpoint that is mission-critical and even a second of downtime would cause insane manual workload.

You have an endpoint that needs to have as little end-user latency as physically possible.

These are all real examples from my work in fintech. The solution to all of them is a few dollars in cloud functions in Lambda and Lambda@Edge. Now imagine having to provision, orchestrate and maintain dozens of dedicated hosts all by yourself, simply for these handful scripts. The expense of employee time alone would make it absolutely idiotic to go that route and if anything this would equal massive scale day - but even worse, because you didn't even plan for it beforehand.

I work in fintech (banking and payments) and most of our clients aren’t even allowed to host on aws; they have to go for a local provider because aws doesn’t have a hosting hub in their country. Not sure what part you work in but this has never been a problem in the past 20 years with just servers, switches, load balancers etc.

I prefer aws over metal for these kind of setups, but for many other cases I definitely do not; just a dedi or a vps with docker or k8s and/or something like openfaas is enough for almost all startups and beyond. Making it literally impossible to make mistakes like OP. And when needed maybe failover or load balancing.

I don’t know about that, the hedge fund I work for is US-based and AWS can be fully SEC and FINRA compliant.

We also have a few dedicated servers, but mostly only for infrequently accessed data and logging that doesn’t need to be highly available.

I really can’t understand why this argument keeps coming up. Different solutions for different usecases. Yet anytime Kubernetes or cloud functions are discussed people come in and go like “hurr durr my single Hetzner dedicated server can do all of that and doesn’t have these problems”

Because some people here do exactly the same thing with aws/cloud. Not you but many here treat aws like it’s the thing you should use ‘because scaling and failover and omg 1 sec downtime’. And then the stories like OP showing that it is dangerous and my experience that most who do this are overpaying and could do with a hetzner server or, better even, a $5/mo vps, even if they do it right (which they are not, generally from all setups I have seen).

I agree with you though on the case by case.

Obviously the billing mechanism (or, lack of control) is an issue but that doesn't negate the whole concept. They could (and should) easily fix this by implementing hard caps to activate if so required.

Nobody says there isn't any use case for AWS. The point is that "the cool kids" like to start their side projects on AWS. Nothing about it is mission critical. I guarantee you op is not working on some Fintech stuff.

How's that bad? Isn't this how we all learned? By playing with cool modern technology?

I feel like learning how to configure bird, bind, haproxy, rdbms clusters, redundant MXs, etc. has taught me far more than spinning up equivalent virtual infrastructure on a public cloud would have. Getting to play with the underlying FOSS technologies is far more rewarding to me than using the commercialized versions on someone else's stack. Plus it helps me evaluate and architect my products for competing clouds, rather than my "knowledge and understanding" being tied to one particular vendor's offerings & lexicon. (Another advantage of knowing the underpinnings of a cloud is that it makes reading post-mortems[1] with a morning coffee so much more enjoyable.)

Assuming we're talking about side-projects/hobbyist development: when I'm doing that I want as few variable expenses as possible, and they usually don't require the purported benefits of the cloud. If such a project needs to scale: I'll bolt it onto a cloud at that point, or sprinkle in specific services to address the pain points.

[1]: https://tritondatacenter.com/blog/manta-postmortem-7-27-2015

> I feel like learning how to configure bird, bind, haproxy, rdbms clusters, redundant MXs, etc. has taught me far more than spinning up equivalent virtual infrastructure on a public cloud would have.

I agree with this, but the choice of what other individuals play with is not ours to make. There's still countless of youngsters working on bare metal, for whatever it's worth, they just grow up in a world with a lot of existing abstractions now.

> Assuming we're talking about side-projects/hobbyist development: when I'm doing that I want as few variable expenses as possible, and they usually don't require the purported benefits of the cloud. If such a project needs to scale: I'll bolt it onto a cloud at that point, or sprinkle in specific services to address the pain points.

100%, but the choice is free for anyone themselves to make. OP is furthering his knowledge and experience in a very specialised and highly sought after sector. Maybe he'll come to the same conclusion at some point, or maybe he comes from bare metal and has different perspectives now and thinks that redundancy is always awesome.

Pretty easy, OP sounds exactly like a guy who would have been perfectly fine with a single (Hetzner) server. Then these arguments popup and they are most of the time right. I agree with you, that it always depends on the use. However, hurr durr Hetzner Server seems to be the more reasonable choice here (once again).

OP here.

Hetzner provides incredibly beefy machines for a good price.

I’ve been managing 2 machine cluster setup in the past using Hetzner and it worked fine.

Unfortunately I have had an incident with one machine going bad unexpectedly(RAM issue) and I was forced to spend whole day setting up a new one.

Why? If you had the choice between complete redundancy and infinite scaling by default while having almost zero work, or using a dedicated server that you need to configure and constantly maintain, what would you choose?

If with one thing, a single mistake can result in a $5k bill, and with the other I have a guaranteed fixed bill of 50 bucks a month, then unless I have enough money to burn, the choice is crystal clear. Maybe you wipe your butt with 100$ bills. Others aren't that lucky.

Exactly; if it is ‘the same’ but with a guarantee of max 50$/mo then sure, I would pick that, but that is not the case, also not with caps. So it is an entirely different case and usually the 50$ option actually brings you very far without any financial risk outside that $50/mo.

The choice is only crystal clear until AWS etc implement hard caps to activate if needed. Then having some Lambda functions is not only exponentially cheaper than a whole dedicated server and by default highly available and redundant, but also equally as safe financially wise.

> You have an endpoint that needs to have as little end-user latency as physically possible.

If you need the lowest possible latency then you should definetly do it yourself. Lambda, even with provisioned concurrency has a very high latency...

If the solution you're building is business-critical then you already have a problem day one: you need high-availability. That means you need at least two servers. As soon as you add that additional server your troubles have begun. Doesn't really much matter whether you add 1 server to your setup or 100.

Your problems compound if you have to consider disaster recovery (DR) and need an offsite location you can failover to. Now you have to contend with data replication and all its concerns.

So even if your application only has a few dozen users things can get quite complex quite fast. AWS makes it simple, practically push-button. Deploying in multiple Availability Zones (AZ) is a no-brainer, and it isn't horribly difficult to deploy in multiple regions, complete with data replication. All while having no servers to configure and maintain. No containers to contend with, no Kubernetes to fight with. To me it's the most stupid-simple solution available today and it's ridiculously cheap! At least so long as you test before you deploy for world exposure!

I completely agree with you. I really suspect some people have never seen themselves how insanely, mind-bogglingly powerful AWS is.

Try doing this [0] with your dedicated servers in under 10 minutes.

And then try attaching highly-available scripts/cloud functions and dozens of different integrated functionality in seconds.

And then try setting up good Network ACL, firewalls, route tables, NAT gateways, load balancers with a few clicks.

... and people here are seriously suggesting that instead using AWS would equal a massive forced scaling operation. Lol.

[0]: https://i.ibb.co/TKmB9HX/image.png

Sure, but not all of us are working at tiny startups or on personal projects. Many of us here work for large companies that have real scale already and are trying to solve those problems.

Why does everyone assume everyone else works at tiny startups?

At that point you're already at day 1000 and already know what scale you need for the project so none of this conversation is relevant. You already have hit the point you need to build it a certain way.

I've never worked at a startup, until last year I never worked at a company with less than 100k employees.

> With all due respect and no offense intended, your perspective sounds a lot like "I've never attempted to scale so I can't understand the problems"

With all due respect, I don't think you've ever actually put together a local cluster.

A simple 4 machine k8s cluster sitting literally on dirt in my basement can scale out to the equivalent of thousands of dollars of AWS spend a month. I broke even on the initial purchase outlay for my workloads in less than a year.

The problems are almost never scaling the web servers. The problem is scaling the infrastructure that those servers need to be useful.

Generally - your DB is the first pain point, your network is the next.

If you can run it in a container, that service probably isn't the bottleneck for scaling, it's going to be whatever is providing the persistent disk for that service, and the network between the two.

Both of those things happen to also be fairly expensive to scale in the cloud as well.

I have a k8s cluster in my basement (concrete floor, not dirt.) It's fine for dev work, but unsuitable for production. The real problem is redundant, reliable internet: cable internet upstream bandwidth sucks after 20 - 40 megabits/sec, and can't get fiber here yet.

This is fair.

I've used both comcast and google fiber while running my basement cluster. I think on google fiber I would have said production usage would be fine - I would pretty reliably get 1gbs down and ~800mbps up.

I wouldn't have redundant net, but the upload speeds are fine.

On Comcast... Comcast is really kind of a joke. Even on expensive business plans they cap upstream at 35mbps in my area (down is 1gbps) and the fastest way to get throttled to actually try using that upstream capacity.

Comcast does give me redundant failover to 5g by default now, which I guess is nice, but it's not worth much when I can't send a ton of data over the line anyways.

So point taken - if you're going to have to rent a room in a major metro for fiber access anyways, you probably want to be paying for cloud services (sadly - you'll pay a lot, since outbound traffic costs are high, and if 30mbps upstream isn't doing it for you, you're gonna blow through the free 100gb fast)

> With all due respect, I don't think you've ever actually put together a local cluster.

I have, and still do. It's neither redundant nor highly available. The power source isn't, the internet connection isn't and it's also located in my basement and not multiple regions.

> The problems are almost never scaling the web servers. The problem is scaling the infrastructure that those servers need to be useful.

There you go..

> Both of those things happen to also be fairly expensive to scale in the cloud as well.

No. [0]

[0]: https://i.ibb.co/TKmB9HX/image.png

With regards to your image... so what? Virtual networking equipment is virtual - I too can spin up hundreds of subnets for nothing.

My internal transfer costs are also.... drumroll... zero. (ok - technically, at some point I bought a 10gbs switch and 500ft of ethernet cable)

Lets talk about what it costs you put data back out onto the net, or into a different region. Then lets compare long term storage costs.

Because that shit is only cheap in the "hobbiest" range. Outside of that, outbound data and storage is fucking expensive in the cloud.

> I too can spin up hundreds of subnets for nothing.

Really? In completely different physical locations with inter-region routing and NAT gateways for each region that route into a redundant load balancer? Then you're right and don't need AWS any more. However, I'd suspect your costs would not be very affordable either.

> Lets talk about what it costs you put data back out onto the net, or into a different region.

If you're seriously trying to compare your residential internet connection with something like this I'm not going to go there..

> Then lets compare long term storage costs.

Same logic applies.

You do realize we're agreeing?

I'm saying the things that are hard to do yourself aren't fucking cheap in the cloud either.

My counter point (and the one you keep dodging) is that the things that are EASY to do yourself aren't fucking cheap in the cloud, and they scale out just fine for the vast majority of non-Saas businesses.

I realize, but I still disagree with one of your points:

> My counter point (and the one you keep dodging) is that the things that are EASY to do yourself aren't fucking cheap in the cloud, and they scale out just fine for the vast majority of non-Saas businesses.

And I don't even disagree with it for its inherent correctness, but because nobody ever said you absolutely have AWS for something that doesn't need it. Most users here tend to discard cloud platforms for even the most obvious and suitable use-cases, and keep on iterating how their single dedicated server can do all of it. I'm also not a big fan of telling someone they don't need to use AWS if it's clearly a side hobby project because that is how we all learned.. by playing with cool modern technology for fun.

The real issue is super simple.. AWS just needs a hard cap functionality so stuff like this can't happen.

> I'm also not a big fan of telling someone they don't need to use AWS if it's clearly a side hobby project because that is how we all learned.. by playing with cool modern technology for fun.

I guess I don't really see the appeal of putting a hobby project into a space where a simple mistake can literally cost you thousands of dollars, with very little recourse.

I get to play with the vast majority of the fun tech - I just choose to do it on a k8s cluster I own, with extremely predictable costs, and frankly - more scaling capacity than I think most businesses need.

> Most users here tend to discard cloud platforms for even the most obvious and suitable use-cases, and keep on iterating how their single dedicated server can do all of it.

And this isn't really my stance - I'm absolutely not advocating for a single dedicated server (fuck - even my hobby instance is HA, 3 machines with a network lb, on a 5 hour UPS, with a redundant network. I personally don't care as much about the redundant network, but comcast provides it free to business customers in my area anyways through 5g equipment).

But I do think most (and I mean 75%+) businesses can spend 10k on a single server rack and go for literal years without problems.

If your business is not selling software as a service - don't buy into the bullshit marketing.

Even if your business is selling software as a service, think about it first. Small businesses tend to get sweetheart deals on costs in AWS, GCP, and Azure because their usage costs for those companies amortizes out to roughly zero, and because they know damn well that they'll end up paying through the nose if they make it to mid-size (or even just screw up once or twice - two fuck ups like the one here and you'd have bought yourself a decent server rack!)

> It's neither redundant nor highly available. The power source isn't, the internet connection isn't and it's also located in my basement and not multiple regions.

For a lot of projects this doesn’t matter. The cost of downtime might be lower than the cost of high availability and occasional downtime is OK.

I'm not trying to dismiss the value there is in having highly scalable cloud architectures available to the projects that need them, but a huge proportion of projects will never need them.

Most projects don't even actually suffer for modest downtime, although you can achieve AWS-comparable downtime even without AWS/Google/Azure-style "cloud" architecture.

It can make a huge difference in operating and engineering costs to know which bucket your project fits in. Good architectural foresight can even let you smoothly move from one to other if unexpected growth or a pivot indicate that it's warranted.

Engineering is about understanding the scope/scale of problems, not about being dogmatic or getting caught up in problems that don't apply.

I was thinking more of when the unpatched Ubuntu 12.04 is exploited via an SSH 0day and the server is used to host a Citibank phishing website.

Most people don't need to scale and even running 5x redundant servers is cheaper than a comparable cloud solution.

Running Lambda, I get a million calls per month for free. Then it's 20 cents per million calls.

Just curious - have you really researched cloud solutions or did you just compare the price of hosting EC2 instances in AWS vs having your own server? Because that's not what cloud is about.

> Running Lambda, I get a million calls per month for free. Then it's 20 cents per million calls.

And you are overpaying by a significant chunk. My raspberry pi - the old cheap one - can handle 10 million requests a day without breaking a sweat. If I push it, I can get up around 90 million requests a day without too much effort (it's only about 1 request/ms)

I really don't think most devs understand how fucking cheap hardware that's comparable to these services is.

Now - you might be using a host of other valuable features that your cloud provider gives you (things like edge servers near your customers, or truly significant outbound network traffic flows, or a very robust multi-region setup, or disk backing of some sort, etc).

But generally speaking - you ARE overpaying for cpu cycles in the cloud. It's not really up for debate.

Of course you are paying more for AWS than for a damn raspberry pi.

You say this like it's obvious that AWS is better.

There are certainly cases where AWS can be better (ease of edge networks and multi-region availability come to mind)

But outside of a very small set of cases (most of which are when companies are victims of their own success - which is actually a wonderful problem to have) what's the compelling reason to actually use AWS if "Of course I am paying more" for it?

In my opinion, if you can still run your db on a single machine - you don't fucking need the cloud yet. That covers a pretty large chunk of businesses.

Most of the "cloud" is convincing business that could self-host their entire stack on a raspberry pi in a basement that they should be spending thousands on cloud compute costs a year.

Fuck - the most damning evidence is simply how much fucking money these companies are making from upselling you cpu cycles that they're getting mostly for free. For the 4th quarter of 2021, amazon reported a profit of 5.2 billion on ~18 billion in revenue from AWS.

So I'm actually in the process of doing Lambda versus persistent costing right now for a new project, where it's heavily load-based and very spiky, but the work on each packet of information is actually very lightweight. The tricky part here in AWS is not Lambda, which is pretty reasonable in general--the pitfalls I'm seeing are around data storage. DynamoDB is stealthily very expensive, either provisioned or on-demand. Right now I'm converging on lambdas for compute and RDS/S3 for storage, but what really makes AWS shine for this use case more than anything else is SQS.

SQS is so good and that there isn't a great, easily deployed (sorry, RabbitMQ) option for exactly what it does is a real push towards AWS. (GCP Cloud PubSub is close enough, but I know AWS way better than I do GCP.) If I didn't need this, and I felt confident managing RabbitMQ as a queueing solution, I don't think AWS would be a compelling solution because $60 in Hetzner nodes, in a HA configuration, could do a lot of work.

(If this thing works I'll need an on-prem solution anyway, so I'll probably have to build that too--but that's "good problems to have".)

DynamoDB is not required to use cloud functions. You can either use regular RDS like you said, Aurora or even just your own EC2-based cluster (all of these you can attach to the VPC the functions are attached to as well) and there's a lot of nice developments going on like Cloudflare D1.

Totally agree on SQS, knowing PubSub as well I'd say they're pretty much on the same level. All the interconnectedness is where cloud platforms shine

To clarify, I mentioned DynamoDB because it has a unit-costs model rather than an amortized-over-time cost model, and when you can harness that (and then focus on a simplified COGS model) you can do very well.

NATS and NSQ are better than SQS if you can deploy them yourself. ZeroMQ is a great option as well, but it probably will require some major brain changes, since it doesn't fit most people's mental model of what a queuing server looks like.

ZeroMQ is too weird for me, yeah. NATS JetStream and NSQ are both promising but their durability options are unclear and while I have a background in devops/system architecture, I'm looking at building something whose entire staff is literally only me and so while I'm sure I can deploy it, I'm not sure I can effectively run it. SQS will do what I expect it to do, and that's pretty powerful.

disclosure: I work at Synadia

We do have a hosted version of NATS called NGS that is multi-cloud, multi-geo and is really easy to set up https://synadia.com/ngs

Your pricing is really very fair, especially since it's based on resource consumption and not per-request. One suggestion: put your pricing right on the home page. I'm starting to wonder why I am running it myself!

And, you probably know a little bit about NATS, too, since, you know, you wrote it! :)

https://nats.io/support/

I think NATS is an excellent, and very easy to run and deploy. Pretty much has a "just works" attribute I like to assign to boring technology

And? I don't think you are aware of how cheap metal is these days.

Plus the OP has already decided that free tiers don't work.