News for hackers. Stuff that matters.
Re:Mistakes I've Made in AWS by sokoloff on Saturday September 11, 2021 @06:54AM (#28490905)
IMO, AWS isn’t competing on cheaper-in-dollars-per-byte but rather in faster and cheaper for your engineering team. If your engineering team is free (as you might decide your time is on a hobby/side project), it’s harder to make the case (I still run my side projects there though), but when they can make the ops team half wearing AWS badges, that offsets a lot of lone-item markups.
Re:Mistakes I've Made in AWS by ratww on Saturday September 11, 2021 @07:47AM (#28491158)
> faster and cheaper for your engineering team.

I really wonder how true that is. Sure, for things like S3 or RDS it’s indeed easier, but for most other things I find AWS either very limiting or extremely arcane.

Even “simple” things like Lambda underdeliver, just this week we run into problems using it with VPC, for example. ElasticBeanstalk was another one, fine-ish for simplistic things but problematic with the smallest customization, also lots of undocumented and undebuggable quirks, like breaking if you use UTF-8 characters in your commit messages, for example.

Of course, we now have the problem where some people, both seniors and juniors, only know or only ever worked with AWS, which makes the assertion that it is “faster and cheaper” correct, but is worrisome, as lots of people are not being taught what used to be the basics 10 or 20 years ago.

Re:Mistakes I've Made in AWS by thinkharderdev on Saturday September 11, 2021 @04:37PM (#28495700)
I recently switched jobs. At my old company the dev team basically had carte blanche to setup their own AWS infra without any real restrictions (there were some but very few). It was nice, I almost never had to ask anybody outside my team to do anything to unblock us (at least from an infra standpoint).

In my new job we also use AWS for everything BUT I haven't used the AWS cli once. I don't even have credentials. Basically, their is a platform team which is responsible for running a k8s cluster (or clusters really) and some other common infrastructure which is all running on bare EC2 (no RDS, no EKS, we do use S3 for blob storage but that's about it). And I have to say it is pretty amazing. No more dealing with arcane IAM rules, or trying to figure out how to string together a chain of lambdas to do some sort of complex orchestration task).

I've really come to appreciate the model of using k8s as your "cloud platform." Having a dedicated team that manages the k8s clusters and makes sure they are elastically scalable and reliable. Everyone else is just deploying stuff to kubernetes. They could decide to move everything to a colo tomorrow and I would have to change exactly nothing about how I do my job.

Re:Mistakes I've Made in AWS by zrail on Saturday September 11, 2021 @08:13AM (#28491325)
Curious what your issue was with Lambda and VPCs. We use that combo all the time where I work and it's fine as long as you have the IAM roles correct.
Re:Mistakes I've Made in AWS by Ancapistani on Saturday September 11, 2021 @01:17PM (#28493735)
Ah, that’s the issue, at least for me - IAM roles are pretty nuanced, and it’s difficult to understand all that’s happening.

I’m working extensively with EventBridge now, and their “security” docs mix “what can access EventBridge” with “what EventBridge can access”. Also, different AWS services all seem to have different requirements - e.g, some are role-based, some are service-based, and some are resource-based. It gets complicated very quickly.

Re:Mistakes I've Made in AWS by selfhoster11 on Saturday September 11, 2021 @04:39PM (#28495723)
Lambda had some gotchas around the warm-up time the last time I used it to implement something. We had to have some extra workarounds to prevent the functions from going "cold".
Re:Mistakes I've Made in AWS by ratww on Saturday September 11, 2021 @09:20AM (#28491756)
Don't get me wrong, I'm a Lambda fan. But unlike EC2/EBS/etc, talking to some AWS services from a Lambda that's inside a VPC requires additional infra and you have to pay for the egress. In the end it just wasn't worth the price for us. It was a bad surprise money-wise.
Re:Mistakes I've Made in AWS by rualca on Saturday September 11, 2021 @08:21AM (#28491375)
> If your engineering team is free (..), it’s harder to make the case (...), but when they can make the ops team half wearing AWS badges, that offsets a lot of lone-item markups.

I have to call bullshit on this claim.

Let's look at the facts. With AWS there are only two scenarios: either you go with the classic "VMs provided by a cloud provider" which is represented by EC2, or you go with hosted services and higher level abstractions like AWS's serverless offerings.

Regarding the EC2, AWS offers absolutely no operational advantage over any other cloud provider, at the expense of being far more expensive. Also, CloudFormation/CDK is arguably far worse and outright developer-hostile than any configuration-as-code alternative. This comparison makes even less sense if we look into AWS' containerization offering, which is either half-baked (ECS) or an afterthought that lags behind alternatives (EKS).

Then we have the higher level abstractions of AWS' managed services and serverless options. Price-gouging runs rampant on this domain, and arguably demands much more training and man-hours to become effective at running production services when compared with just running your own services. This scenario entails higher costs and the only arguments that any ops team can muster revolve around sunk cost and vendor lock-in.

Re:Mistakes I've Made in AWS by sokoloff on Saturday September 11, 2021 @10:09AM (#28492099)
The price gouging services make sense if they avoid you having to hire additional employees. That's the benefit of any managed service provider: they can run it more efficiently than you can (once you add all the people, supervisors of those people, people to cover when those people are on vacation, etc).

It's a way to shift people costs to IT operational expenses. If you don't do that, it's more expensive. If you do, it can easily be less expensive. I'm pretty sure we're at the point where it's less expensive because developers are waiting minutes rather than weeks [or more] for TechOps actions to happen (we were on-prem previously [and I ran TechOps]). That saves time and changes the way you think about TechOps changes. If they're lengthy, you make choices that avoid changes in TechOps. If they're fast, you make choices that make the most sense for the product and customer.

Re:Mistakes I've Made in AWS by rualca on Saturday September 11, 2021 @10:37AM (#28492295)
> The price gouging services make sense if they avoid you having to hire additional employees.

But the fact is that it doesn't. It's another service that needs training/experience to develop and operate. Arguing about these hypothetical savings is just a veiled appeal to the sunk cost fallacy and vendor lock-in I've mentioned.

> I'm pretty sure we're at the point where it's less expensive because developers are waiting minutes rather than weeks [or more] for TechOps actions to happen (we were on-prem previously [and I ran TechOps]).

I'm not sure this scenario is remotely realistic for the past decade or so, specially after the inception of containerization. Even in bare metal deployments anyone can get multiple databases configured and going in a matter of minutes.

Re:Mistakes I've Made in AWS by sokoloff on Saturday September 11, 2021 @12:35PM (#28493315)
Containers don’t get you more host machines racked or more disk shelves added to the SAN. On-prem, it solves configuration within a (nearly) fixed scale which, if that’s your only problem, is great. If you’re in your own DC/colo, there’s more advantages to moving out than containers can provide alone.

I literally can’t afford to invest to the level that AWS can to run operations. AWS bandwidth is incredibly expensive right up until the point where you or a neighbor is getting a DDoS attack that Amazon just “handles” for you. My customers don’t care where we’re hosted and won’t pay extra for either on-prem or cloud-hosted. They just want it to be up and transparent. For us, AWS is cheaper/faster all-in. That’s not true for everyone and, if it’s not, please don’t use it.

Re:Mistakes I've Made in AWS by rualca on Saturday September 11, 2021 @02:14PM (#28494341)
> Containers don’t get you more host machines racked or more disk shelves added to the SAN.

That's immaterial to the discussion, and reads like a non-sequitur. You want a service. You deploy the service in your infrastructure. If necessary, you scale your infrastructure to meet demand. That's it. If you want to spin up a database instance, just do it. With containerization that takes between minutes and seconds.

And to drive the point home, in case you're not aware, AWS is not the only cloud provider that offers horizontal autoscaling. Some small providers even sell it out of the box, both through their Kubernetes offerings and/or through their own APIs.

Also, the sales brochure for managed services mentions scalability and reliability, and in the case of AWS also global deployments, but the truth of the matter is that it costs a hefty premium and in most cases it's totally irrelevant.

So, pointing out databases in practical terms means close to nothing.

> I literally can’t afford to invest to the level that AWS can to run operations.

And that's perfectly fine because a) AWS really is not full proof (see the latest outage of AWS's US-WEST-2 region which might have single-handedly dropped AWS's reliability to only 99.5 this year), b) operating your own infra already gets you plenty of 9s easily, c) the theoretical difference in the 9s you get and the 9s that are advertised by AWS is more often than not totally irrelevant to the usecases you need to meet.

To sum it up, you may argue all you want about how AWS's Rolls Royce is far superior than any car in the market, but the truth of the matter is that the vast majority has all their needs decisively met and even surpassed by running any other cloud provider's Ford hatchback.

Re:Mistakes I've Made in AWS by noogle on Saturday September 11, 2021 @08:47AM (#28491565)
But it doesn't save much engineering time. You now need to manage those services, and the high mark-up means you need to invest effort in scaling things up/down.

Yes, it took me about a week to learn to set-up a Postgresql high availability cluster. But now it saves me $4,000 per MONTH for each of our 10 databases.

And if you are using EC2 instances, AWS saves very little effort compared to bare metal.

Re:Mistakes I've Made in AWS by Daishiman on Saturday September 11, 2021 @06:49PM (#28496829)
So did you send your database logs to a centralized logging system? Did you set up roles and keys to access those systems? Are your roles integrated with the rest of your permissions system? Do you have a perf dashboard where you can see the real-time usage of your DBs? Have you already rehearsed updating your database version?

The thing is that it's totally possible that you learned everything needed to set up the cluster, in my experience most database systems that aren't set up by a professional DBA will sooner or later hit a configuration or maintenance snag. Once that happens, you're pretty much totally on your own and for critical systems that downtime is going to cost you more than the costs of your infra.

So you either need to have a DBA on retainer if you're serious about data integrity, or you pay management costs which means your system was set by literally the most expert people on the planet in the area.

If you're running a cluster where performance is the highest priority and downtime and maintenance isn't a huge issue because you have a nice decent maintenance window and enough dev cycles to spend on staying up to date, for sure, go for it.

But in my experience, if you care more about a system staying up, good managed infra is so much more reliable that it's not even a question.

Re:Mistakes I've Made in AWS by noogle on Sunday September 12, 2021 @08:54AM (#28500703)
> So did you send your database logs to a centralized logging system?

Not needed.

> Did you set up roles and keys to access those systems?

Yes.

> Are your roles integrated with the rest of your permissions system?

Not needed. Access to the DB is limited to a very small circle, by design. Postgresql's mechanisms suffice.

> Do you have a perf dashboard where you can see the real-time usage of your DBs?

htop/task manager and Postgresql's internal monitoring have so far been sufficient.

> Have you already rehearsed updating your database version?

Indirectly yes, although AFAIK (happy to learn otherwise) major version update are also not supported by RDS.

This has been working for years now, with actual paying customers (so "in production"). It's always about opportunity-cost and diminishing returns: RDS costs about as 3 developers. The benefit the product gets from 3 more developers far outweighs the benefit of another "9" in the SLA.

Even if all of the savings would have gone to pay for a DBA, we'd still get a better deal: an in-house DBA will focus on addressing our specific needs. RDS also has a serious opportunity cost since it prohibits using extensions that are very useful for us, forcing us to spend time on workarounds.

And frankly, it's not like RDS is even that reliable. We had multiple incidents where the database stopped responding or slowed down to a crawl for several hours, and we had now way to debug this (since we cannot access the server itself). They also don't seem to handle anything about Postgresql tuning.

For the customer, the app slowing down (because RDS has a very limited IO bandwidth) is an outage (and is much more frequent than some system error).

tl;dr: Many people can meet their needs with a Ford, and living on the streets to pay for the extra features of a Ferrari is not a good tradeoff for them.

EDIT: line spacing