I have a personal dev cloud on AWS that has:

* AWS MSK (Kafka server) with redundant instances across 3 AZs

* Aurora Posgres instance with redundancy across 3 AZs

* Multiple Lambdas for incoming messages from MQTT to Kafka

* IoT/MQTT and about a dozen iot devices sending telemetry continuously (every second)

* 20+ services with nearly 40 tasks, running on ECS

* All the supporting route53, certificates, VPCs etc. for the above

* A bunch of miscellaneous stuff I've forgotten that's used by other developers.

* All deployed via TF

_THAT_ all costs well under a grand. You're doing it wrong.

I think that's unfair. Your setup involving more distinct services than his has little bearing on the amount of data you're working with. Our setup is a lot simpler than yours (no postgres, lambda, mqtt) but deals with presumably a lot more data (~800Gib daily) so it costs an order of magnitude more than yours.

Absolutely. My setup is, as stated, a dev instance. We have a substantially larger prod setup, but I can't get into the details about production scale on a public forum. That said, even the dev instance is moving a substantial amount of data; nothing like 800GB/day, but within an order of magnitude.

And yes, bandwidth costs can be brutal, especially if you try to build a hybrid solution that uses heavy services that live outside of AWS.

That said, the above comment didn't get into enough detail to get an accurate read on their bandwidth or CPU/Memory/Storage utilization, but if they can run it in a lab for $5/mo (which I doubt, since that barely covers power), then it can't be much, so I think it's fair to assume it's at least back-of-the-napkin similar scale.

Most all of those services are pay for use. Which means listing out the services you use means nothing. The price would be based on your level of usage. I have a bunch of lambdas setup in AWS and it's basically free because at the moment, I'm not running them.

Our ECS containers all run continuously processing Kafka messages. Dev has variable throughput, but it's hundreds of messages per second at least. The containers all use between 256MB and 2048MB commits (they're stream processors, so they don't hammer the memory like a stateful monolithic service would).

My point was just that it doesn't have to cost over $1k on Amazon to run a combination of services which apparently CAN run in a private lab for $5/mo, however that is calculated. I can't run my full stack even without load (due to memory commit) on my 32GB, 3 year old Xeon workstation, and that would cost a lot more than $5/mo at any provider I am aware of.

Although he doesn't mention API Gateway in his list, I used to design AWS solutions for people, and it was by far the biggest mistake I saw them make cost-wise: It can get scary-expensive in a hurry, especially if you aren't careful at your cloud architecture and really understand what all you'll be paying for.

Difference is probably that you run real microservices that have small footprint and you can binpack them on very small instances together with very very small aurora cluster.

If you wrote exactly the numbers of instances and types -> we could have a real comparison then.

Which is most likely the case 99% of times people decry a technology on HN

Is it really the same, or 'similar, but worse'? AWS is expensive, but multi-zone redundancy is expensive anywhere and even worse: it's not available everywhere.

That also means that a side-project that doesn't need that redundancy would be a mismatch in terms of requirements anyway. On the other hand, if you need a multi zone automatically available system, you're running out of choices really fast.

Based upon outages when a single AWS zone goes offline, a whole hell of a lot more than side projects aren't taking advantage of multi-zone redundancy.

I think what many fail to realize is that - surprise surprise! It isn't that simple. Many think "if my zone goes out, I'll just migrate to a different zone!" Nope! AWS doesn't have the capacity for it - as we've seen many times, AWS can't seem to handle the migratory load when a zone goes out. Sure, the other zones don't really die, but good luck migrating your workload to them.

But when touting multi zones, no one ever mentions this little nugget of information.

At my previous work we often were able to saturate all similar instance types in given region..

All AWS had to say was “we are working hard on providing additional capacity”.

The same often happens on black friday, where companies scale up their platforms just in case because there might be no capacity on AWS.

We have that 'issue' with many of our Spark workloads where there isn't any of our desired capacity available as spot, but we have a baseline reserved up front instances for anything realtime anyway so with a bit of planning it's a non-issue.

It does cost money, but then again, so does not running certain processes. The trick becomes calculating the intersection at which point the costs outweigh the benefits, and that calculation applies everywhere.

If you're in a single AZ you're not in multiple AZ. Migrating between AZs isn't multi-zone either. Running at 130% capacity in three AZs, that is multi-AZ (to us, in our availability configuration). If an AZ goes down (which in some regions we use has happend 0 times) we lose about 30% capacity, but since that's our margin of scaling anyway we can keep going as-is, even if there was no 15% additional capacity available in the remaining AZs.

Some sort of manual active-standby configuration really doesn't require AWS or a Cloud, that stuff is the same 90's implementation it has always been and practically boils down to attaching your RAID1 USB HDDs from one PC to another PC and booting that bad boy up as 'failover'. (yes, that's an example, and yes it's an extreme one)

If you have capacity planning, and you plan accordingly, you take service provider limits into account, just like you would with anything else. Having two power feeds into a distribution warehouse doesn't help much if neither can't handle 100% of the load in an industrial park. So while having two feeds might seem 'redundant' to a single tenant or customer, it's only really redundant if either can supply all the demand of all connected customers.

The same applies to fiber connections, plenty of fake-redundant connections that are suggested by customers to be 'redundant' turn out to end up at the same PoP and if the PoP goes down your redundant fibers are worthless. In the same logistics distribution scenario, your trucks can't deliver goods if the destination warehouse itself is offline, and now you need redundant warehouses.

That's obviously a weird thing to do at smaller scales, but the fact remains that AWS having an AZ go down is only a small piece of the puzzle, and only really a problem if you didn't plan for it appropriately.

Automatic multi-zone failover to survive az cashes without outages seems to be commonly based on optimism, as people don't effectively drill for it regularly. There are always bugs in it that need ironing out. So people just pay the price in operating costs, keeping replicas running on other AZs, and the resulting sw/distributed system/infra complexity, but but don't actually make the last steps to reap any payoff. It's because single-az outages are rare enough and that still leaves relatively common aws-wide outages you face and your self inflicted footgun outages. And drilling for the az failover can be scary and risky if you haven't done it before and are already in production.

AZ redundancy is like storage backups or backup power, if you don't regularly test it you don't have it.

This is true, but I will argue again that I am proficient enough with AWS to deploy a full stack with 10+ lambdas, DNS, certs, RDS, step functions, etc for < 5 dollars a month. You need to understand what to use and what not to. Most of that bill is from the nat gateway/private VPC, which you really dont need

How? Last time I looked the smallest RDS for MySQL came at 11 USD.

RDS for < 5 dollars a month?

>Deployed few services on ecs, nat gateway, shared elb, certificate and route53.

>Total cost 30$ per day ~900$ per month.

Ummm... I think you're doing something incorrectly. The biggest line items on that list are ELB (assuming you mean Application Load Balancer/ALB) and NAT Gateway. But with that setup, I could maybe imagine $100/month with on-demand plans. Not $900.

How can you say that with ECS on the list? You have no idea how much compute is being used.

Well he said he can replicate it with a $5/mo setup elsewhere.

How much compute can you rent anywhere for $5/mo? How much compute can you get even buying the hardware and amortizing over its useful lifespan for $5/mo (~$300 total)?

Reading over the post, it says $5 in their "local lab". Which probably means using existing or older hardware, so the $5 is probably electricity costs or similar.

And it sounded more like a generalisation anyway. ;)

ECS can get up there if you're backing with large ec2 instances or something

either way my biggest problem with services like AWS isn't really cost but that you end up using all of their SDKs and services out of convenicance and it becomes hard to track down or move away from.. it spiders out of control so fast that sometimes its better to spend an extra few weeks doing things the harder way that's not always an issue though espcially for startups

NAT is $1/day and ELB is 60cents/day. So those can't (although those costs excludes data transfer...) be the biggest line items.

NAT is $1/day *PER* AZ *PER* VPC.

Managed NAT Gateways can be a very substantial cost depending on your architecture. I believe the recommended VPC layout has 3 AZs so you're paying close to $100/month for a single VPC just for NAT.

Ding ding ding, ppl here dont take to account HA and disaster preventions you sometimes have to abaid to due to security certifications and regulations!

So your comparison with your $5 local lab is not just less "convenient"!

Do you really need multi-multi-region failover redundancy? Do you really need multiple NATs in every region (it's easy to share one NAT Gateway across AZs, especially so if a region is only servicing DR)? Your numbers still sound high.

x $1 + y $0.60 to make up $30 is a _lot_ of regions/AZs.

Yes the $5/month lab setup compared to a globally redundant service.

what exactly do you set up for 5 bucks a month on your local lab? Sounds more like you’re comparing what lightsail offers, which incidentally costs the same even on aws?

If you want a simple sight with just core aws, you can set one up on aws on elastic beanstalk for a few bucks a month. Add an rds for 15 a month. Why will you needlessly overcomplicate the system and then cry foul?

Tbh 5$ was to just show the order of magnitude.

Even if it was 50$ per month, its still alot cheaper.

And the cluster we are talking here about is a raspberry box with 10 mashines each 2cpu 8gb, running k8s, replicated storage, my own domain and certificate, metalb load balancers, and few other things.

Its not as redundant as AWS, but it never broke in few years of constant work. Chances of all of those machines to break at the same time are astronomically low in normal circumstances.

If I spread the cost of that setup over two years of reservation, the monthly cost will be a lot less than 50$.

$900 a month? The only thing in that list that could come close might be ECS? Are you using ECS anywhere?

If you have high compute needs I've had good experience with that and it's cheap. You can put a huge box behind it. I did that (I have ATT fiber to the house) and for the $7/month I could still use all my deploy templates and AWS CLI commands and code to do stuff.

I tried this with an AMD home machine (AMD 5900x / 64 GB ram / nvme) and it worked surprisingly well as did a dell server (beefier). I put ubuntu on them, registered them, and they came right up in the plane.

> So the knowledge required to setup that and hardware is worth around 695$ per month.

That's an extremely cheap price to pay if you had to hire someone to do it.

Yeah a single infra engineer in my city gets paid $15k/mo total comp.

I get the pain if you’re at “garage with shoestring budget” stage but $1k/mo isn’t even worth the time to set up a billing alarm.

We have around 20 API GW, at least 20 dynamodb tables and more than 30 lambdas, several queues and opensearch replicated across 2 environments (dev and prod) plus additional programming environments (cloud9) and we barely come close to half of what you you are budgeting here for. Your math is wrong.

How big are those Opensearch clusters ?

Lets take a small production cluster of 4x r6g.large (default proposal for prod) (0.167$/h) deployed in VPC in private subnet (x2 for HA)

That’s 16$ per day for instances only and 2.5$ per day for nat-gateways per day. So 18.5$ per day and ~550$ per month.

Thats ofcourse if you disable all metrics all logs all alerting etc and it does not include Volumes cost, because its highly dependent on the domain.

AWS is expensive because you get alot of administrative work out of the box (snapshots, restores, etc). Tho sometimes have to still fix in the source. Ive personally pushed quite a few fixes to OpenSearch source, coz version 1.0.0 had a lot of issues.

$900/month – whether thats too little or too much entirely depends on the company's revenue/margins. I've seen companies that do not care about AWS spend. It makes close to zero difference to them and just thinking about the cost costs more. On the other spectrum, I've seen companies that move to bare metal servers. A couple of servers and that's all they need. $120/month from $2000/month and it mattered to them.

So yeah, what you're saying is sort of meaningless without understanding the business type/stage/revenue/costs.

That $5 a month figure - does that include your internet connection? I have a far more complicated AWS setup that costs me about $100/mo. I'm assuming the extra costs you are incurring are for data?

5$ per month is pretty impressive. What hardware would you be running this solution on? Typically if you run hardware in a co-located data center there's a fee for that, is that included in the 5$?

What did the hardware cost, and based on depreciation, how long will it last before requiring replacement? Is that also included in the 5$ you are quoting here?