News for hackers. Stuff that matters.
Re: by electroly on Friday February 09, 2024 @07:40PM (#39322275)
> The markup cost of using RDS (or any managed database) is worth it.

Every so often I price out RDS to replace our colocated SQL Server cluster and it's so unrealistically expensive that I just have to laugh. It's absurdly far beyond what I'd be willing to pay. The markup is enough to pay for the colocation rack, the AWS Direct Connects, the servers, the SAN, the SQL Server licenses, the maintenance contracts, and a full-time in-house DBA.

https://calculator.aws/#/estimate?id=48b0bab00fe90c5e6de68d0...

Total 12 months cost: 547,441.85 USD

Once you get past the point where the markup can pay for one or more full-time employees, I think you should consider doing that instead of blindly paying more and more to scale RDS up. You're REALLY paying for it with RDS. At least re-evaluate the choices you made as a fledgling startup once you reach the scale where you're paying AWS "full time engineer" amounts of money.

Re: by steveBK123 on Friday February 09, 2024 @08:00PM (#39322393)
RDS pricing is deranged at the scales I've seen too. $60k/year for something I could run on just a slice of one of my on-prem $20k servers. This is something we would have run 10s of. $600k/year operational against sub-$100k capital cost pays DBAs, backups, etc with money to spare.

Sure, maybe if you are some sort of SaaS with a need for a small single DB, that also needs to be resilient, backed up, rock solid bulletproof.. it makes sense? But how many cases are there of this? If its so fundamental to your product and needs such uptime & redundancy, what are the odds its also reasonably small?

Re: by macNchz on Friday February 09, 2024 @08:26PM (#39322573)
> Sure, maybe if you are some sort of SaaS with a need for a small single DB, that also needs to be resilient, backed up, rock solid bulletproof.. it makes sense? But how many cases are there of this?

Most software startups these days? The blog post is about work done at a startup after all. By the time your db is big enough to cost an unreasonable amount on RDS, you’re likely a big enough team to have options. If you’re a small startup, saving a couple hundred bucks a month by self managing your database is rarely a good choice. There’re more valuable things to work on.

Re: by tw04 on Friday February 09, 2024 @11:00PM (#39323450)
>By the time your db is big enough to cost an unreasonable amount on RDS, you’re likely a big enough team to have options.

By the time your db is big enough to cost an unreasonable amount on RDS, you've likely got so much momentum that getting off is nearly impossible as you bleed cash.

You can buy a used server and find colocation space and still be pennies on the dollar for even the smallest database. If you're doing more than prototyping, you're probably wasting money.

Re: by theptip on Friday February 09, 2024 @11:50PM (#39323686)
That’s just another way of saying the opportunity cost isn’t worth paying to do the migration.

Optionality and flexibility are extremely valuable, and that is why cloud compute continues to be popular, especially for rapidly/burstily growing businesses like startups.

Re: by latch on Saturday February 10, 2024 @12:41AM (#39323912)
I don't mean to pick on your specific comments, but I find these analysis almost always lack a crucial perspective: level of knowledge. This is the single biggest factor, and it's the hardest one to be honest about. No one wants to say "RDS is a good choice . . . because I don't know how nor have I ever self managed a database."

If you want a different opportunity cost, get people with different experience. If RDS is objectively expensive, objectively slow, but subjectively easy, change the subject.

Re: by pcl on Saturday February 10, 2024 @02:12AM (#39324247)
> No one wants to say "RDS is a good choice . . . because I don't know how nor have I ever self managed a database."

I don't think that's accurate. I've self-managed databases, and I still think that RDS is compelling for small engineering teams.

There's a lot to get right when managing a database, and it's easy to screw something up. Perhaps none of the individual parts are super-complicated, but the cost of failure is high. Outsourcing that cost to AWS is pretty compelling.

At a certain team size, you'll end up with a section of the team that's dedicated to these sorts of careful processes. But the first place these issues come up is with the database, and if you can put off that bit of organizational scaling until later, then that's a great path to choose.

Re: by maccard on Saturday February 10, 2024 @08:08AM (#39325903)
I disagree here. This falls apart when you zoom out one step. I'm perfectly capable of managing a database. I'm also capable of maintaining load balancers, redis, container orchestrators, Jenkins, perforce, grafana, Loki, Oncall, individually. But each of those has the high chance of being a distraction from what our software actually does.

Its about tradeoffs, and some tradeoffs are often more applicable than others - getting a ping at 7am on a Sunday because your ec2 instance filled it's drive up with logs and your log rotation script failed because it didn't have a long enough retey is a problem I'm happy to outsource when I should be focusing on the actual app.

Re: by theptip on Sunday February 11, 2024 @02:21PM (#39337726)
Lack of expertise in some particular technology is simply another opportunity cost. I can learn how to operate a production DB at scale (I have racked servers and run other production workloads) but as cofounder/CTO in a startup is that the best use of my time?

If the cost of a hosted DB is going to sink the company, then of course, I will figure it out and run it myself. But it’s not, for most startups. And therefore that knowledge isn’t providing much leverage.

Starting an AI company with deep expertise in training models - that is an example of knowledge providing huge leverage. DB tech is not in this bucket for most businesses.

Re: by graemep on Saturday February 10, 2024 @05:12AM (#39325020)
People do not really understand the value of the former. Even dealing with financial options (buy/sell and underlying) which are a pure form of it, people either do not understand the value, or do so in a very abstract way they do not intuit.
Re: by matwood on Saturday February 10, 2024 @05:50AM (#39325197)
Good point. And, since you brought up financials, you also see this when people use a majority of their savings to lump sum pay off a mortgage. They take an overweighted view of saving on interest and, IMO, underweight the flexibility of liquidity.
Re: by graemep on Saturday February 10, 2024 @07:17AM (#39325617)
On the other hand cloud platforms can be hard to migrate off, which is very much taking away options.
Re: by macNchz on Saturday February 10, 2024 @12:44AM (#39323928)
In the small SaaS startup case, I’d say the production database is typically the most critical single piece of infra, so self hosting is just not a compelling proposition unless you have a strong technical reason where having super powerful database hardware is important, or a team with multiple people who have sysadmin or DBA experience. I think both of those cases are unusual.

I’ve been the guy managing a critical self-hosted database in a small team, and it’s such a distraction from focusing on the actual core product.

To me, the cost of RDS covers tons of risks and time sinks: having to document the db server setup so I’m not the only one on the team who actually knows how to operate it, setting up monitoring, foolproof backups so I don’t need to worry that they’re silently failing because a volume is full and I misconfigured the monitoring, PITR for when someone ships a bad migration, one click HA so the database itself is very unlikely to wake me at 3am, blue/green deploys to make major version upgrades totally painless, never having to think about hardware failures or borked dist-upgrades, and so on.

Each of those is ultimately either undifferentiated work to develop in-house RDS features that could have been better spent on product, or a risk of significant data loss, downtime, or firefighting. RDS looks like a pretty good deal, up to a point.

Re: by remus on Saturday February 10, 2024 @02:15AM (#39324264)
I like fiddling with databases, but I totally agree with this. Unless you really need a big database and are going to save 100k+ per year by going self managed then RDS or similar just saves you so much stress. We've been using it for the best part of 10 years and uptime and latency have consistently been excellent, and functionality is all rock solid. I never have to think about it, which is just what I want from something so core to the business.
Re: by matwood on Saturday February 10, 2024 @05:46AM (#39325182)
I am good at databases (have been a DBA in the past), and 100% agree with this. RDS is easy to standup and get all the things you mentioned, and not have to think about again. If we grow to the point where the overhead is more than a FT DBA, awesome. It means we are successful, and are fortunate to have options.
Re: by rnts08 on Saturday February 10, 2024 @08:33AM (#39326011)
Unfortunately there are so many people and teams who thinks that simply running their databases on RDS means that they're backed up, highly-available and can be easily load balanced, upgraded, partitioned, migrated and so on which is simply not the case with the basic configuration.

RDS is a great choice, for prototyping and only for production if you know what you're doing when setting it up.

FWIW, this is common in all cloud deployments, people assume that running something "severless" is a magical silver bullet.

Re: by macNchz on Saturday February 10, 2024 @10:26AM (#39326789)
Well…just using the defaults when creating an RDS Postgres in the console give you an HA cluster with two read replicas, 7 days of backups restorable to any point in time, automatic minor version upgrades, and very easy major upgrades. So unless you start actively unchecking stuff those are not entirely invalid assumptions.
Re: by optymizer on Saturday February 10, 2024 @08:27AM (#39325980)
I agree, but I also classify some of these as "learn them once and you're all set".

Maybe it takes you a month the first time around and a week the 10th time around. First product suffers, the other products not so much. Now it just takes a week of your time and does not require you to pay large AWS fees, which means you are not bleeding money

I like to set up scrappy products that do not rack up large monthly fees. This means I can let them run unprofitable for longer and I don't have to seek an investor early, which would light up a large fire under everyone's butts and start influencing timelines because now they have the money and want a return asap.

I'll launch a week later - no biggie usually. I could have come up with the idea a month later, so I'm still 3 weeks early ;)

It doesn't work for all projects, obviously, but I've seen plenty of SaaS start out with a shopping spree, then pay monthly fees and purchase licenses for stuff that they could have set up for free if they put some (usually not a lot) effort into it. When times get rough, the shorter runway becomes a hard fact of life. Maybe they wouldn't have needed a VC and could have bootstrapped and also survived for longer.

Re: by macNchz on Saturday February 10, 2024 @09:23AM (#39326326)
Learning it all is what gave me an appreciation for RDS! I’ve self managed a number of Postgres and MySQL databases, including a 10TB Postgres cluster with all of the HA and backup niceties.

While I generally agree as far as initial setup time goes, I favor RDS because I can forget about it, whereas the hand rolled version demands ongoing maintenance, and incurs a nonzero chance of simple mistakes that, if made, could result in a 100% dataloss unrecoverable scenario.

I’m also mostly talking about typical, funded startups here, as opposed to indie/solo devs. If you’re flying solo launching a tiny proof of concept that may only ever have a few users, by all means run it yourself if you’d like, but if you’ve raised money to grow faster and are paying employees to iterate rapidly searching for PMF…just pay for RDS and make sure as much time as possible is spent on product features that provide actual business value. It starts at like $15/month. The cost of simply not being laser-focused on product is far greater.

Re: by crazygringo on Saturday February 10, 2024 @02:00PM (#39329170)
> you've likely got so much momentum that getting off is nearly impossible as you bleed cash.

Databases are not particularly difficult to migrate between machines. Of all the cloud services to migrate, they might actually be the easiest, since the databases don't have different API's that need to be rewritten for, and database replication is a well-established thing.

Getting off is quite the opposite of nearly impossible.

Re: by viraptor on Friday February 09, 2024 @08:57PM (#39322777)
Lots of cases. It doesn't even have to be a tiny database. Within <1TB range there's a huge number of online companies that don't need to do more than hundreds of queries per second, but need the reliability and quick failover that RDS gives them. The $600k cost is absurd indeed, but it's not the range of what those companies spend.

Also, Aurora gives you the block level cluster that you can't deploy on your own - it's way easier to work with than the usual replication.

Re: by steveBK123 on Friday February 09, 2024 @09:25PM (#39322955)
Once you commit to more deeply Amazon flavored parts of AWS like Aurora, aren't you now fairly committed to hoping your scale never exceeds the cost-benefit tradeoff?
Re: by nemothekid on Saturday February 10, 2024 @12:16AM (#39323796)
If my scale exceeds the cost benefit tradeoff, then I will thank God/Allah/Buddah/Spaghetti Monster.

These questions always sound flawed to me. It's like asking won't I regret moving to California and paying high taxes once I start making millions of dollars? Maybe? But that's an amazing problem to have and one that I may be much better equipped to solve.

If you are small, RDS is much cheaper, and many company killing events, such as not testing your backups are solved. If you are big and you can afford a 60K/yr RDS bill than you can make changes to move on-prem. Or you can open up excel and do the math if your margins are meaningfully affected by moving on-prem.

Re: by pclmulqdq on Saturday February 10, 2024 @12:30AM (#39323865)
I assume that you do that math on all your new features too, right? The calculation of how much extra money they will bring in?

On some level, AWS/GCP/California relies on you doing this calculation for the things that you can do it on easily (the savings of moving away), while not doing this calculation on things where it's hard to do (new development). That way, you can pretend that your new features are a lot more valuable than the $Xk/year you will save by moving your infra.

Re: by nemothekid on Saturday February 10, 2024 @07:29AM (#39325690)
>The calculation of how much extra money they will bring in?

Yes, I've done the math. The piece you are missing is, saving money on infra will bring in $0 new dollars. There is a floor to how much money I can save. There is no ceiling to how much money the right feature can bring in. Penny pinching on infra, especially when the amount of money is saved is less than the cost of an engineer is almost always a waste of time while you are growing a company. If you are at the point where you are wasting 1x,2x,3x of an engineers salary of superflous infrastructure - then congratulations you have survived the great filter for 99% of startups.

>That way, you can pretend that your new features are a lot more valuable than the $Xk/year you will save by moving your infra.

Finding product market fit is 1000x harder than moving from RDS to On-prem. If you haven't solved PMF, then no amount of $Xk/year in savings will save you from having to shut down your company.

Re: by pclmulqdq on Saturday February 10, 2024 @03:21PM (#39329832)
I am well aware of the math on that. Also, switching to faster infra can be a surprising benefit to your revenue, by the way, if it makes your app feel nicer.

The thing is, most features, particularly later in the life of a company, don't have an easy-to-measure revenue impact, and I suspect that many features are actually worth $0 of revenue. However, they cost money to implement (both in engineering time and infra), making them very much net negative value propositions. This is why Facebook and Google can cut tons of staff and lose nothing off their revenue number.

Also, there's a bit of a gambling mentality here which is that a feature could be worth effectively infinite revenue (ie it could be the thing that gives you PMF), so it's always worth doing over things with known, bounded impact on your bottom line. However, improving your efficiency gives you more cracks at finding good features before you run out of money.

Re: by matwood on Saturday February 10, 2024 @05:55AM (#39325213)
Agree. "What if you're wildly successful and get huge?" Awesome, we'll solve the problem then. The other part is what if AWS was a part of becoming successful? IE, it freed my small team from having to worry all that much about a database and instead focused on features.
Re: by callalex on Friday February 09, 2024 @10:58PM (#39323440)
If you’re paying list price at scale you are doing it very wrong.
Re: by tw04 on Friday February 09, 2024 @11:02PM (#39323465)
Sure, but if you're paying anywhere near list price for your on-prem hardware at scale you're also doing it wrong. I've never seen a scenario where Amazon discounts exceed what you would get from a hardware or software vendor at the same scale.
Re: by osigurdson on Saturday February 10, 2024 @01:03AM (#39323994)
Interesting how cloud services are sold like used cars.
Re: by rswail on Saturday February 10, 2024 @02:20AM (#39324287)
It's more interesting how cloud services are sold like any other consumables or corporate services.

No one runs their own electricity supply (well until recently with renewables/storage), they buy it as a service, up to a pretty high scale before it becomes more economic to invest the capex and opex to run your own.

Re: by viraptor on Friday February 09, 2024 @09:55PM (#39323118)
Or you're realistic about what you're doing. Will you ever need to scale more than 10x? And on the timescales where you do grow over 10x, would it be better to reconsider/re-architect everything anyway?

I mean, I'm looking after a 4 instance Aurora cluster which is great feature wise, is slightly overprovisioned for special events, and is more likely to shrink than grow 2x in the next decade. If we start experiencing any issues, there's lots of optimisations that can be still gained from better caching and that work will be cheaper than the instance size upgrade.

Re: by zmgsabst on Friday February 09, 2024 @09:56PM (#39323126)
…no?

There’s still a defined cost to swapping your DB code over to a different backend. At the point where it becomes uneconomical, you’re also at a scale you can afford rewriting a module.

That’s why we have things like “hexagonal architecture”, which focus on isolating the storage protocol from the code. There’s an art to designing such that your prototype can scale with only minor rework — but that’s why we have senior engineers.

Re: by rswail on Saturday February 10, 2024 @02:22AM (#39324296)
Aurora supports standard Postgres clients.

So moving to/from Aurora/RDS/own EC2/on-prem should be a matter of networking and changing connection strings in the clients.

Your operational requirements and processes (backup/restore, failover, DR etc) will change, but that's because you're making a deliberate decision weighing up those costs vs benefits.

Re: by gregw2 on Saturday February 10, 2024 @07:24AM (#39325664)
Pro tip side note:

You can use DNS to mitigate the pain of changing those connection strings, decoupling client change management from backend change process, or if you had foresight, not having to change client connection strings at all.

Re: by viraptor on Saturday February 10, 2024 @10:17PM (#39332386)
Nope, nope, nope! When you change DNS entries, they will take effect at some point in the future when the cache expires and when your app decides to reconnect. (Possibly after a restart) At that point, why not be sure and change the config?

I mean, DNS change can work, but when you're doing that one-in-years change, why risk the extra failure modes.

Re: by DiggyJohnson on Friday February 09, 2024 @09:22PM (#39322942)
The US DoD for sure.
Re: by amluto on Friday February 09, 2024 @10:20PM (#39323253)
I have a small MySQL database that’s rather important, and RDS was a complete failure.

It would have cost a negligible amount. But the sheer amount of time I wasted before I gave up was honestly quite surprising. Let’s see:

- I wanted one simple extension. I could have compromised on this, but getting it to work on RDS was a nonstarter.

- I wanted RDS to _import the data_. Nope, RDS isn’t “SUPER,” so it rejects a bunch of stuff that mysqldump emits. Hacking around it with sed was not confidence-inspiring.

- The database uses GTIDs and needed to maintain replication to a non-AWS system. RDS nominally supports GTID, but the documented way to enable it at import time strongly suggests that whoever wrote the docs doesn’t actually understand the purpose of GTID, and it wasn’t clear that RDS could do it right. At least Azure’s docs suggested that I could have written code to target some strange APIs to program the thing correctly.

Time wasted: a surprising number of hours. I’d rather give someone a bit of money to manage the thing, but it’s still on a combination of plain cloud servers and bare metal. Oh well.

Re: by blantonl on Saturday February 10, 2024 @08:45AM (#39326085)
replication to non-AWS systems. "simple" extension problems importing data into RDS because of your custom stuff lurking in a mysqldump

Sounds like you are walking massive edge

Re: by ehnto on Saturday February 10, 2024 @12:14AM (#39323786)
> Sure, maybe if you are some sort of SaaS with a need for a small single DB, that also needs to be resilient, backed up, rock solid bulletproof.. it makes sense? But how many cases are there of this?

Very small businesses with phone apps or web apps are often using it. There are cheaper options of course, but when there is no "prem" and there are 1-5 employees then it doesn't make much sense to hire for infra. You outsource all digital work to an agency who sets you up a cloud account so you have ownership, but they do all software dev and infra work.

> If its so fundamental to your product and needs such uptime & redundancy, what are the odds its also reasonably small?

Small businesses again, some of my clients could probably run off a Pentium 4 from 2008, but due to nature of the org and agency engagement it often needs to live in the cloud somewhere.

I am constantly beating the drum to reduce costs and use as little infra as needed though, so in a sense I agree, but the engagement is what it is.

Additionally, everyone wants to believe they will need to hyperscale, so even medium scale businesses over-provision and some agencies are happen to do that for them as they profit off the margin.

Re: by graemep on Saturday February 10, 2024 @05:32AM (#39325116)
A lot of my clients are small businesses in that range or bigger.

AWS and the like are rarely a cost effective option, but it is something a lot of agencies like, largely because they are not paying the bills. The clients do not usually care because they are comfortable with a known brand and the costs are a small proportion of the overall costs.

A real small business will be fine just using a VPS provider or a rented server. This solves the problem of not having on premise hardware. They can then run everything on a single server, which is a lot simpler to set up, and a lot simpler to secure. That means the cost of paying someone to run it is a lot lower too as they are needed only occasionally.

They rarely need very resilient systems as they amount of money lost to downtime is relatively small - so even on AWS they are not going to be running in multiple availability zones etc.

Re: by neeleshs on Saturday February 10, 2024 @12:46AM (#39323931)
Out of curiosity, who is your onprem provider?
Re: by kunley on Saturday February 10, 2024 @05:17AM (#39325045)
RDS is not so bulletproof as advertised, and the support is first arrogant then (maybe) helpful.

People pay for RDS because they want to believe in a fairy tale that it will keep potential problems away and that it worked well for other customers. But those mythical other customers also paid based on such belief. Plus, no one wants to admit that they pay money in such irrational way. It's a bubble

Re: by AtlasBarfed on Saturday February 10, 2024 @05:49PM (#39330943)
Plus aws outright lie to us about zero downtime upgrades.

Come time for force major upgrade shoved down our throat? Downtime, surprise, surprise

Re: by thelastparadise on Saturday February 10, 2024 @07:48AM (#39325810)
> $600k/year operational against sub-$100k capital cost pays DBAs, backups, etc with money to spare.

One of these is not like the others (DBAs are not capex.)

Have you ever considered that if a company can get the same result for the same price ($100K opex for RDS vs same for human DBA), it actually makes much more sense to go the route that takes the human out of the loop?

The human shows up hungover, goes crazy, gropes Stacy from HR, etc.

RDS just hums along without all the liabilities.

Re: by AaronM on Saturday February 10, 2024 @09:55AM (#39326555)
Not only that, you can't just have one DBA. You need a team a them, otherwise that person is going to be on call 24/7, can never take a vacation, etc. Your probably looking at a minimum of 3.
Re: by tpetry on Saturday February 10, 2024 @10:54AM (#39327080)
And when you have performance issues you still need a DBA. Because RDS only runs your database. It is up to you to make it fast.
Re: by icedchai on Saturday February 10, 2024 @02:11PM (#39329257)
You'll need an engineer with database skills, not a dedicated DBA. I haven't seen a small company with a full time DBA in well over a decade. If you can learn a programming language, you can learn about indexes and basic tuning parameters (buffer pool, cache, etc.)
Re: by vasco on Friday February 09, 2024 @07:45PM (#39322303)
That's a huge instance with an enterprise license on top. Most large SaaS companies can run off of $5k / m or cheaper RDS deployments which isn't enough to pay someone. The amount of people running half a million a year RDS bills might not be that large. For most people RDS is worth it as soon as you have backup requirements and would have to implement them yourself.
Re: by sgarland on Friday February 09, 2024 @08:17PM (#39322515)
> Most large SaaS companies can run off of $5k / m or cheaper RDS

Hard disagree. An r6i.12xl Multi-AZ with 7500 IOPS / 500 GiB io1 books at $10K/month on its own. Add a read replica, even Single-AZ at a smaller size, and you’re half that again. And this is without the infra required to run a load balancer / connection pooler.

I don’t know what your definition of “large” is, but the described would be adequate at best at the ~100K QPS level.

RDS is expensive as hell, because they know most people don’t want to take the time to read docs and understand how to implement a solid backup strategy. That, and they’ve somehow convinced everyone that you don’t have to tune RDS.

Re: by rswail on Saturday February 10, 2024 @02:29AM (#39324322)
If you're not using GP3 storage that provides 12K minimum IOPS without requiring provisioned IOPS for >400GB storage, as well as 4 volume striping, then you're overpaying.

If you don't have a reserved instance, then you're giving up potentially a 50% discount on on-demand pricing.

An r6i.12xl is a huge instance.

There are other equivalents in the range of instances available (and you can change them as required, with downtime).

Re: by sgarland on Saturday February 10, 2024 @09:44AM (#39326465)
> GP3... as well as 4 volume striping

For MySQL and Postgres, RDS stripes across four volumes once you hit 400 GiB. Doesn't matter the type.

The latency variation on gp3 is abysmal [0], and the average [1] isn't great either. It's probably fine if you have low demands, or if your working set fits into memory and you can risk the performance hit when you get an uncached query.

12K IOPS sounds nice until you add latency into it. If you have 2 msec latency, then (ignoring various other overheads, and kernel or EBS command merging) the maximum a single thread can accomplish in one second is (1000 msec / 1 sec / 2 msec) = 500 I/O. Depending on your needs that may be fine, of course.

> If you don't have a reserved instance, then you're giving up potentially a 50% discount on on-demand pricing.

True, of course. Large customers also don't pay retail.

> An r6i.12xl is a huge instance.

I mean, it goes well past that to .32xl, so I wouldn't say it's huge. I work with DBs with 1 TiB of RAM, and I'm positive there are people here who think those are toys. The original comment I replied to said, "large SaaS," and a .12xl, as I said, would be roughly adequate for ~100K QPS, assuming no absurdly bad queries.

[0]: https://www.percona.com/blog/performance-of-various-ebs-stor...

[1]: https://silashansen.medium.com/looking-into-the-new-ebs-gp3-...

Re: by electroly on Friday February 09, 2024 @07:50PM (#39322330)
Definitely--I recommend this after you've reached the point where you're writing huge checks to AWS. Maybe this is just assumed but I've never seen anyone else add that nuance to the "just use RDS" advice. It's always just "RDS is worth it" full stop, as in this article.
Re: by Aeolun on Friday February 09, 2024 @08:00PM (#39322401)
To some extend that is probably true, because when you’ve built a business that needs a 500k/year database fully on RDS it’s already priced into your profits, and switching to a self-hosted database will seem unacceptably risky for something that works just fine.
Re: by groestl on Saturday February 10, 2024 @02:55AM (#39324437)
> it’s already priced into your profits

Assuming you have any. You might not, because of AWS.

Re: by sroussey on Friday February 09, 2024 @08:45PM (#39322706)
I mean, just use supabase instead. So much easier than RDS. Why even deal with AWS directly? Might as well have a Colo if you need AWS.
Re: by dzikimarian on Saturday February 10, 2024 @04:32AM (#39324848)
>Most large SaaS companies can run off of $5k / m or cheaper RDS deployments which isn't enough to pay someone.

After initial setup, managing equivalent of $5k/m RDS is not full time job. If you add to this, that wages differ a lot around the world, $5k can take you very, very far in terms of paying someone.

Re: by renewiltord on Friday February 09, 2024 @07:51PM (#39322335)
You don't get the higher end machines on AWS unless you're a big guy. We have Epyc 9684X on-prem. Cannot match that at the price on AWS. That's just about making the choices. Most companies are not DB-primary.
Re: by sgarland on Friday February 09, 2024 @08:21PM (#39322541)
I think most people who’ve never experienced native NVMe for a DB are also unaware of just how blindingly fast it is. Even io2 Block Express isn’t the same.
Re: by sroussey on Friday February 09, 2024 @08:47PM (#39322716)
Most databases expressly say don’t run storage over a network.
Re: by amluto on Friday February 09, 2024 @10:45PM (#39323374)
To be fair, most networked filesystems are nowhere near as good as EBS. That’s one AWS service that takes real work to replicate on-prem.

OTOH, as noted, EBS does not perform as well as native NVMe and is hilariously expensive if you try. And quite a few use cases are just fine on plain old NVMe.

Re: by tpetry on Saturday February 10, 2024 @03:13AM (#39324502)
Thats because EBS is a network block device and not a network filesystem - that would be EFS. And with network block devices you get the same perf and better compared to EBS.
Re: by renewiltord on Friday February 09, 2024 @08:29PM (#39322604)
Yes. We have it 4x striped on those same machines. Burns like lightning.
Re: by Cacti on Saturday February 10, 2024 @12:19AM (#39323807)
I have one of those. It’s so fast I don’t even know what to do with it.
Re: by sgarland on Friday February 09, 2024 @08:35PM (#39322639)
The only problem is it hides all of the horrible queries. Ah well, can’t have it all.
Re: by icelancer on Saturday February 10, 2024 @02:15AM (#39324263)
Ha, I did just the same thing - and also optimized for an extremely fast per-thread CPU (which you never get from managed service providers).

The query times are incredible.

Re: by ndriscoll on Saturday February 10, 2024 @11:45AM (#39327665)
Funny enough, the easiest way to experience this is probably to do some performance experimentation on the machine you code on. If it's a laptop made in the last few years, the performance you can get out of it knowing that it's sipping on a 45W power brick with probably not great cooling will make you very skeptical of when people talk about "scale".
Re: by infecto on Friday February 09, 2024 @09:24PM (#39322952)
The problem you have here is by the time you reach the size of this DB, you are on a special discount rate within AWS.
Re: by jacurtis on Friday February 09, 2024 @10:56PM (#39323426)
Discount rates are actually much better too on the bigger instances. Therefore the "sticker price" that people compare on the public site is no where close to a fair comparison.

We technically aren't supposed to talk about pricing publically, but I'm just going to say that we run a few 8XL and 12Xl RDS instances and we pay ~40% off the sticker price.

If you switch to Aurora engine the pricing is absurdly complex (its basically impossible to determine without a simulation calculator) but AWS is even more aggressive with discounting on Aurora, not to mention there are some legit amazing feature benefits by switching.

I'm still in agreeance that you could do it cheaper yourself at a Data Center. But there are some serious tradeoffs made by doing it that way. One is complexity and it certainly requires several new hiring decisions. Those have their own tangible costs, but there are a huge amount of intangible costs as well like pure inconvenience, more people management, more hiring, split expertise, complexity to network systems, reduce elasticity of decisions, longer commitments, etc.. It's harder to put a price on that.

When you account for the discounts at this scale, I think the cost gap between the two solutions is much smaller and these inconveniences and complexities by rolling it yourself are sometimes worth bridging that smaller gap in cost in order to gain those efficiencies.

Re: by jq-r on Saturday February 10, 2024 @04:25AM (#39324811)
> but I'm just going to say that we run a few 8XL and 12Xl RDS instances and we pay ~40% off the sticker price.

Genuinely curious, how do you that?

We pay a couple of million dollars per year and the biggest spend is RDS. The bulk of those are 8xl and 12xl as you mention and we have a lot of these. We do have savings plans, but those are nowhere near 40%.

Re: by hardolaf on Saturday February 10, 2024 @08:03AM (#39325883)
Yeah 40% seems like a pipedream. I was at a Fortune 500 defense firm and we couldn't get any cloud provider to even offer us anything close to that discount if we agreed to move to them for 3-4 years minimum. That org ended up not migrating because it was significantly cheaper to buy land and build datacenters from scratch than to rent in the cloud.
Re: by overstay8930 on Saturday February 10, 2024 @01:50PM (#39329075)
There are basically no discounts in govcloud
Re: by hardolaf on Sunday February 11, 2024 @12:53AM (#39332937)
Defense firms do a lot more than just government work. Also, there are definitely discounts in govcloud when Fortune 500 companies that operate 30+ datacenters start talking to govcloud providers about potentially migrating to their services.
Re: by CubsFan1060 on Saturday February 10, 2024 @09:21AM (#39326309)
At least according to: https://instances.vantage.sh/rds/?selected=db.r6g.16xlarge,d...

It looks like a reserved instance is 35% off sticker price? Add probably a discount and you'd be around 40% off.

Re: by CubsFan1060 on Saturday February 10, 2024 @09:17AM (#39326272)
The new Aurora pricing model helps, and is honestly the only reason we're able to use it. It caps costs: https://aws.amazon.com/blogs/aws/new-amazon-aurora-i-o-optim...
Re: by nyc_data_geek on Friday February 09, 2024 @09:55PM (#39323124)
Some orgs are looking at moving back to on prem because they're figuring this out. For a while it was vogue to go from capex to opex costs, and C suite people were incentivized to do that via comp structures, hence "digital transformation" ie: migration to public cloud infrastructure. Now, those same orgs are realizing that renting computers actually costs more than owning them, when you're utilizing them to a significant degree.

Just like any other asset.

Re: by stingraycharles on Saturday February 10, 2024 @12:13AM (#39323779)
That’s made possible because of all the orchestration platforms such as Kubernetes being standardized, and as such you can get pretty close to a cloud experience while having all your infrastructure on-premise.
Re: by nyc_data_geek on Saturday February 10, 2024 @12:40AM (#39323911)
Yes, virtualization, overprovisioning and containerization have all played a role in allowing for efficient enough utilization of owned assets that the economics of cloud are perhaps no longer as attractive as they once were.
Re: by nextos on Friday February 09, 2024 @10:56PM (#39323429)
Same experience here. As a small organization, the quotes we got from cloud providers have always been prohibitively expensive compared to running things locally, even when we accounted for geographical redundancy, generous labor costs, etc. Plus, we get to keep know how and avoid lock-in, which are extremely important things in the long term.

Besides, running things locally can be refreshingly simple if you are just starting something and you don't need tons of extra stuff, which becomes accidental complexity between you, the problem, and a solution. This old post described that point quite well by comparing Unix to Taco Bell: http://widgetsandshit.com/teddziuba/2010/10/taco-bell-progra.... See HN discussion: https://news.ycombinator.com/item?id=10829512.

I am sure for some use-cases cloud services might be worth it, especially if you are a large organization and you get huge discounts. But I see lots of business types blindly advocating for clouds, without understanding costs and technical tradeoffs. Fortunately, the trend seems to be plateauing. I see an increasing demand for people with HPC, DB administration, and sysadmin skills.

Re: by layoric on Friday February 09, 2024 @11:17PM (#39323549)
> Plus, we get to keep know how and avoid lock-in, which are extremely important things in the long term.

So much this. The "keep know how" has been so greatly avoided over the past 10 years, I hope people with these skills start getting paid more as more companies realize the cost difference.

Re: by lanstin on Friday February 09, 2024 @11:46PM (#39323661)
When I started working in the 1980s (as a teenager but getting paid) there was a sort of battle between the (genuinely cool and impressive) closed technology of IBM and the open world of open standards/interop like TCP/IP and Unix, SMTP, PCs, even Novell sort of, etc. There was a species of expert that knew the whole product offering of IBM, all the model numbers and recommended solution packages and so on. And the technology was good - I had an opportunity to program a 3093K(?) CM/VMS monster with APL and rexx and so on. Later on I had a job working with AS/400 and SNADS and token ring and all that, and it was interesting; thing is they couldn't keep up and the more open, less greedy, hobbyists and experts working on Linux and NFS and DNS etc. completely won the field. For decades, open source, open standards, and interoperability dominated and one could pick the best thing for each part of the technology stack, and be pretty sure that the resultant systems would be good. Now however, the Amazon cloud stacks are like IBM in the 1980s - amazingly high quality, but not open; the cloud architects master the arcane set of product offerings and can design a bespoke AWS "solution" to any problems. But where is the openness? Is this a pendulum that goes back and forth (and many IBM folks left IBM in the 1990s and built great open technologies on the internet) or was it a brief dawn of freedom that will be put down by the capital requirements of modern compute and networking stacks?

My money is on openness continuing to grow and more and more pieces of the stack being completely owned by openness (kernels anyone?) but one doesn't know.

Re: by nyc_data_geek on Saturday February 10, 2024 @12:53AM (#39323964)
Even without owning the infrastructure, running in the cloud without know-how is very dangerous.

I hear tell of a shop that was running on ephemeral instance based compute fleets (EC2 spot instances, iirc), with all their prod data in-memory. Guess what happened to their data when spot instance availability cratered due to an unusual demand spike? No more data, no more shop.

Don't even get me started on the number of privacy breaches because people don't know not to put customer information in public cloud storage buckets.

Re: by hardolaf on Saturday February 10, 2024 @07:51AM (#39325824)
I was part of a relatively small org that wanted us to move to cloud dev machines. As soon as they saw the size of our existing development docker images that were 99.9% vendor tools in terms of disk space, they ran the numbers and told us that we were staying on-prem. I'm fairly sure just loading the dev images daily or weekly would be more expensive than just buying a server per employee.
Re: by nicbou on Saturday February 10, 2024 @08:35AM (#39326025)
Is there a bit of risk involved since the know-how has a will of its own and sometimes gets sick?

If I had a small business with very clever people I'd be very afraid of what happens if they're not available for a while.

Re: by oooyay on Saturday February 10, 2024 @01:33AM (#39324111)
Context: I build internal tools and platforms. Traffic on them varies, but some of them are quite active.

My nasty little secret is for single server databases I have zero fear of over provisioning disk iops and running it on SQLite or making a single RDBMS server in a container. I've never actually run into an issue with this. It surprises me the number of internal tools I see that depend on large RDS installations that have piddly requirements.

Re: by DeathArrow on Saturday February 10, 2024 @05:56AM (#39325220)
>making a single RDBMS server in a container

On what disk is the actual data written? How do you do backups, if you do?

Re: by BirAdam on Saturday February 10, 2024 @08:18AM (#39325933)
In most setups like this, it’s going to be spinning rust with mdadm, and MySQL dumps that get created via cron and sent to another location.
Re: by dvfjsdhgfv on Saturday February 10, 2024 @11:35AM (#39327533)
The problem with single instance is that while performance-wise it's best (at least on bare metal), there comes a moment when you simply have too much data and one machine can't handle. Your your scenario, it may never come up, but many organizations face this problem sooner or later.
Re: by oooyay on Saturday February 10, 2024 @03:38PM (#39329984)
I agree, my point is that clusters are overused. Most applications simply don't need them and it results in a lot of waste. Much of this has to do with engineers being tasked with an assortment of roles these days, so they obviously opt for the solution where a database and upgrades are managed for them. I've just found that managing a single containers upgrades aren't that big of an issue.
Re: by chii on Saturday February 10, 2024 @02:23AM (#39324302)
i would imagine that cloud infrastructure has the ability for fast scale up, unlike self-owned infrastructure.

For example, how long does it take to rent another rack that you didnt plan for?

And not to mention that the cost of cloud management platforms that you have to deploy to manage these owned assets is not free.

I mean, how come even large consumers of electricity does not buy and own their own infrastructure to generate it?

Re: by tpetry on Saturday February 10, 2024 @03:04AM (#39324469)
Ordering that amount of amount of servers takes about one hour with hetzner. If you truly want a complete rack on your own maybe a few days as they have to do it manually.

Most companies don‘t need to scale up full racks in seconds. Heck, even weeks would be ok for most of them to get new hardware delivered. The cloud planted the lie into everyone‘s head that most companies dont have predictable and stable load.

Re: by rajamaka on Saturday February 10, 2024 @05:27AM (#39325095)
What would be the cost/time of scaling down a rack on Hetzner?
Re: by pinkgolem on Saturday February 10, 2024 @06:04AM (#39325257)
rental period is a month you can also use hetzner cloud, which is still roughly 10x less expensive then aws and that does not take into account the vastly cheaper traffic
Re: by hardolaf on Saturday February 10, 2024 @07:54AM (#39325841)
Most businesses could probably know server needs 6-12 months out. There's a small number of businesses in the world that actually need dynamic scaling.
Re: by pinkgolem on Saturday February 10, 2024 @02:50AM (#39324402)
>I mean, how come even large consumers of electricity do not buy and own their own infrastructure to generate it?

They sure do? BASF has 3 power plants in Hamburg, Disney operate Reedy Creek Energy with at least 1 power plant and I could list a fair bit more...

>For example, how long does it take to rent another rack that you didnt plan for?

I mean, you can also rent hardware a lot cheaper then on AWS. There certainly are providers where you can rent out a rack for a month within minutes

Re: by sseagull on Saturday February 10, 2024 @08:57AM (#39326146)
Some universities also have their own power plants. It’s also becoming more common to at least supplement power on campus with solar arrays.
Re: by gorm on Saturday February 10, 2024 @06:02AM (#39325245)
One other appealing alternative for smaller startups is to run Docker on one burstable vm. This is a simple setup and allows you to go beyond the cpu limits and also scale up the vm.

Might be other alternatives than using Docker so if anyone has tips for something simpler or easier to maintain, appreciate a comment.

Re: by pinkgolem on Saturday February 10, 2024 @02:36AM (#39324354)
Keep in mind, there is an in between..

I would have a hard time doing servers as cheap as hetzner for example including the routing and everything

Re: by jwr on Saturday February 10, 2024 @03:05AM (#39324474)
I do that. In fact I've been doing it for years, because every time I do the math, AWS is unreasonably expensive and my solo-founder SaaS would much rather keep the extra money.

I think there is an unreasonable fear of "doing the routing and everything". I run vpncloud, my server clusters are managed using ansible, and can be set up from either a list of static IPs or from a terraform-prepared configuration. The same code can be used to set up a cluster on bare-metal hetzner servers or on cloud VMs from DigitalOcean (for example).

I regularly compare this to AWS costs and it's not even close. Don't forget that the performance of those bare-metal machines is way higher than of overbooked VMs.

Re: by pinkgolem on Saturday February 10, 2024 @03:21AM (#39324527)
I was more talking about physical backbone connection which hetzner does for you.

We are using hetzner cloud.. but we are also scaling up and down a lot right now

Re: by fgonzag on Saturday February 10, 2024 @04:03AM (#39324712)
You usually just do colocation. The data center will give you a rack (or space for one), an upstream gateway to your ISP, and redundant power. You still have to manage a firewall and your internal network equipment, but its not really that bad. I've used PFsense firewalls, configured by them for like $1500, with roaming vpn, high availability, point to point vpn, and as secure as reasonably possible. After that it's the same thing as the cloud except its physical servers.
Re: by pinkgolem on Saturday February 10, 2024 @04:09AM (#39324735)
i mean, yes.. but you pay for that, and colocation + server deprication in the case i calculated was higher then just renting the servers
Re: by swores on Saturday February 10, 2024 @03:33AM (#39324579)
Could you please explain what you mean by "physical backbone connection", as I can't think of a meaning that fits the context.

If you mean dealing with the physical dedicated servers that can be rented from Hetzner, that's what the person you replied to was talking about being not so difficult.

If you mean everything else at the data centre that makes having a server there worthwhile (networking, power, cooling, etc.) I don't think people were suggesting doing that themselves (unless you're a big enough company to actually be in the data centre business), but were talking about having direct control of physical servers in a data centre managed by someone like Hetzner.

(edit: and oops sorry I just realised I accidentally downvoted your comment instead of up, undone and rectified now)

Re: by pinkgolem on Saturday February 10, 2024 @03:44AM (#39324633)
With "routing" I meant the backbone connection, which is included in the hetzner price.

Aka if I add up power (including backup) + backbone connection rental + server deprication I can not do it for the hetzner price..

That was quite imprecise, sorry about that.

Re: by DeathArrow on Saturday February 10, 2024 @06:02AM (#39325244)
I think no one talked about having physical server on their own premises but colocating servers in a data center or renting servers in a data center.
Re: by swores on Saturday February 10, 2024 @09:01AM (#39326163)
No worries, easy to not foresee every possible way in which strangers could interpret a comment!

But I think that people (at least jwr, and probably even nyc_data_geek saying "on prem") are talking about cloud (like AWS) vs. renting (or buying) servers that live in a data centre run by a company like Hetzner, which can be considered "on prem" if you're the kind of data centre client who has building access to send your own staff there to manage your servers (while still leaving everything else, possibly even legal ownership and therefore deprecation etc. to the data centre owner).

What you're thinking of - literally taking responsibility for running your own mini data centre - I think is hardly ever considered (at least in my experience), except by companies at the extremes of size. If you're as big as Facebook (not sure where the line is but obviously including some companies not AS big as Meta but still huge) then it makes sense to run your own data centres. If you're a tiny business getting less than thousands of website visits a day and where the website (or whatever is being hosted) isn't so important that a day of downtime every now and then isn't a big deal, then it's not uncommon to host from the company's office itself (just using a spare old PC or second hand cheap 1U server, maybe a cheap UPS, and just connected to the main internet connection that people in the office use, and probably managed by a single employee, or company owner, who happens to be geeky enough to think it's one or both of simple or fun to set up a basic LAMP server, or even a Windows server for its oh-so-lovely GUI).

Re: by DeathArrow on Saturday February 10, 2024 @05:59AM (#39325235)
100% agree. People still think that maintaining infrastructure is very hard and requires lot of people. What they disregard is that using cloud infrastructure also requires people.
Re: by tormeh on Saturday February 10, 2024 @07:16AM (#39325610)
When talking about Hetzner pricing, please don’t change the subject to AWS pricing. The two have nothing in common, and intuition derived from one does not transfer to the other.
Re: by dvfjsdhgfv on Saturday February 10, 2024 @11:39AM (#39327581)
> please don’t change the subject to AWS pricing

Why? The only reason I'm using Hetzner and not AWS for several of my own projects (even though I know AWS much better since this is what I use at work) is an enormous price difference in each aspect (compute, storage, traffic).

Re: by KronisLV on Saturday February 10, 2024 @01:54PM (#39329119)
> The two have nothing in common

If all you need are some cloud servers, or a basic load balancer, they are pretty much the same.

If you need a plethora of managed services and don't want to risk getting fired over your choice or specifics of how that service is actually rendered, they are nothing alike and you should go for AWS, or one of the other large alternatives (GCP, Azure etc.).

On the flip side, if you are using AWS or one of those large platforms as a glorified VPS host and you aren't doing this in an enterprise environment, outside of learning scenarios, you are probably doing something wrong and you should look at Hetzner, Contabo, or one of those other providers, though some can still be a bit pricey - DigitalOcean, Vultr, Scaleway etc.

Re: by jwr on Saturday February 10, 2024 @07:11PM (#39331504)
> the two have nothing in common

Well, in my case at least, what they have in common is that I can choose to run my business on one or the other. So it's not about intuition, but rather facts in my case: I avoid spending a significant amount of money.

I (of course) do realize that if you design your software around higher-level AWS services, you can't easily switch. I avoided doing that.

Re: by jumploops on Saturday February 10, 2024 @03:51AM (#39324666)
Funny story time.

I was once part of an acquisition from a much larger corporate entity. The new parent company was in the middle of a huge cloud migration, and as part of our integration into their org, we were required to migrate our services to the cloud.

Our calculations said it would cost 3x as much to run our infra on the cloud.

We pushed back, and were greenlit on creating a hybrid architecture that allowed us to launch machines both on-prem and in the cloud (via a direct link to the cloud datacenter). This gave us the benefit of autoscaling our volatile services, while maintaining our predictable services on the cheap.

After I left, apparently my former team was strong-armed into migrating everything to the cloud.

A few years go by, and guess who reaches out on LinkedIn?

The parent org was curious how we built the hybrid infra, and wanted us to come back to do it again.

I didn't go back.

Re: by smitty1e on Saturday February 10, 2024 @04:52AM (#39324943)
My funny story is built on the idea that AWS is Hotel California for your data.

A customer had an interest in merging the data from an older account into a new one, just to simplify matters. Enterprise data. Going back years. Not even leaving the region.

The AWS rep in the meeting kinda pauses, says: "We'll get back to you on the cost to do that."

The sticker shock was enough that the customer simply inherited the old account, rather than making things tidy.

Re: by hhsectech on Saturday February 10, 2024 @05:17AM (#39325051)
Eh? I've never had a problem moving data out of AWS.

Have people lost the ability to write export and backup scripts?

Re: by interroboink on Saturday February 10, 2024 @05:45AM (#39325174)
My (peripheral) experience is that it is much cheaper to get data in than to get data out. When you have the amount of data being discussed — "Enterprise data. Going back years." — that can get very costly.

It's the amount of data where it makes more sense to put hard drives on a truck and drive across the country rather than send it over a network, where this becomes an issue (actually, probably a bit before then).

Re: by fcarraldo on Saturday February 10, 2024 @09:20AM (#39326300)
AWS actually has a service for this - Snowmobile, a storage datacenter inside of a shipping container, which is driven to you on a semi truck. https://aws.amazon.com/snowmobile/
Re: by xmcqdpt2 on Saturday February 10, 2024 @09:46AM (#39326482)
They do not!

> Q: Can I export data from AWS with Snowmobile? > > Snowmobile does not support data export. It is designed to let you quickly, easily, and more securely migrate exabytes of data to AWS. When you need to export data from AWS, you can use AWS Snowball Edge to quickly export up to 100TB per appliance and run multiple export jobs in parallel as necessary. Visit the Snowball Edge FAQs to learn more.

https://aws.amazon.com/snowmobile/faqs/?nc2=h_mo-lang

Why would they make it convenient to leave?

Re: by fcarraldo on Saturday February 10, 2024 @11:30AM (#39327453)
Oh, TIL! Thanks for correcting me.
Re: by brickteacup on Saturday February 10, 2024 @09:52AM (#39326531)
That's only for data into AWS though, not data out
Re: by Draiken on Saturday February 10, 2024 @05:32AM (#39325121)
The ingress/egress cost is ridiculously high. Some companies don't care, but it is there and I've seen it catch people off guard multiple times.
Re: by varjag on Saturday February 10, 2024 @07:35AM (#39325720)
Oh come on from the description both accounts could be sitting on the same datacenter LAN.
Re: by LadyCailin on Saturday February 10, 2024 @05:35AM (#39325127)
It’s the cost of data egress, which isn’t free.
Re: by mciancia on Saturday February 10, 2024 @06:46AM (#39325472)
But there is no paid egress when we are moving data between account within one region, rigth?
Re: by storyinmemo on Saturday February 10, 2024 @07:33AM (#39325707)
There is. You pay a price for any cross-VPC traffic.
Re: by CubsFan1060 on Saturday February 10, 2024 @09:15AM (#39326257)
This isn't true, at least not anymore.

You can peer two vpc's and as long as you are transferring within the same (real) AZ, it's free: https://aws.amazon.com/about-aws/whats-new/2021/05/amazon-vp...

Even peered VPC's only pay "normal" prices: https://aws.amazon.com/ec2/pricing/on-demand/#Data_Transfer

"Data transferred "in" to and "out" from Amazon EC2, Amazon RDS, Amazon Redshift, Amazon DynamoDB Accelerator (DAX), and Amazon ElastiCache instances, Elastic Network Interfaces or VPC Peering connections across Availability Zones in the same AWS Region is charged at $0.01/GB in each direction."

Re: by Shorel on Saturday February 10, 2024 @09:13AM (#39326245)
Just in network costs, there's a huge asymmetry. Uploading data to AWS is free. Downloading data from them, you have to pay.

When you have enough data, that cost is quite significant.

Re: by mijoharas on Saturday February 10, 2024 @09:50AM (#39326508)
There's a cost for data egress (but not ingress)
Re: by banku_brougham on Saturday February 10, 2024 @09:19AM (#39326291)
Is R2 a sensible option for hosting data? I understand egress is chesp.
Re: by stickfigure on Saturday February 10, 2024 @10:50AM (#39327020)
R2 is great. Our GCS bill (almost all egress) jumped from a few hundred dollars a month to a couple thousand dollars a month last year due to a usage spike. We rush-migrated to R2 and now that part of the bill is $0.

I've heard some people here on HN say that it's slow, but I haven't noticed a difference. We're mainly dealing with multi-megabyte image files, so YMMV if you have a different workload.

Re: by banku_brougham on Saturday February 10, 2024 @11:20PM (#39332618)
awesome. I remember reading about this a while ago, but never tried. Since it has the same API i can imagine its not daunting as a multi-cloud infrastructure.

I guess permissions might be more complex, as in EC2 instance profiles wouldnt grant access, etc.

Re: by stickfigure on Monday February 12, 2024 @11:37AM (#39346914)
Just to make sure nobody is confused by this - R2 has the same API as S3, not GCS. We had to build a simple abstraction around GCS/S3 to perform the migration. But if you're migrating from S3, it's pretty much drop-in. We even use the AWS-provided S3 Java library (especially convenient for making signed URLs).
Re: by hhsectech on Saturday February 10, 2024 @05:16AM (#39325038)
There are two possible scenarios here. Firstly, they can't find the talent to support what you implemented...or more likely, your docs suck!

I've made a career out of inheriting other peoples whacky setups and supporting them (as well as fixing them) and almost always its documentation that has prevented the client getting anywhere.

I personally dont care if the docs are crap because usually the first thing I do is update / actually write the docs to make them usable.

For a lot of techs though crap documentation is a deal breaker.

Crap docs aren't always the fault of the guys implementing though, sometimes there are time constraints that prevent proper docs being written. Quite frequently though its outsourced development agencies that refuse to write it because its "out of scope" and a "billable extra". Which I think is an egregious stance...doxs Should be part and parcel of the project. Mandatory.

Re: by adrianmsmith on Saturday February 10, 2024 @07:30AM (#39325693)
> Quite frequently though its outsourced development agencies that refuse to write it

It's also completely against their interest to write docs as it makes their replacement easier.

That's why you need someone competent on the buying side to insist on the docs.

A lot of companies outsource because they don't have this competency themselves. So it's inevitable that this sort of thing happens and companies get locked in and can't replace their contractors, because they don't have any docs.

Re: by smokel on Saturday February 10, 2024 @05:40AM (#39325149)
I agree that bad documentation is a serious problem in many cases. So much so that your suggestion to write the documentation after the fact can become quite impossible.

If there is only one thing that juniors should learn about writing documentation (be it comments or design documents), it is this: document why something is there. If resources are limited, you can safely skip comments that describe how something works, because that information is also available in code.

(It might help to describe what is available, especially if code is spread out over multiple repositories, libraries, teams, etc.)

(Also, I suppose the comment I'm responding to could've been slightly more forgiving to GP, but that's another story.)

Re: by lazyasciiart on Saturday February 10, 2024 @06:43AM (#39325456)
Unfortunately it’s also possible that e.g the company switched from share point to confluence and lost half the entire knowledge base because it wasn’t labeled the way they thought it was. Or that the docs were all purged because they were part of an abandoned project.
Re: by maxrecursion on Saturday February 10, 2024 @07:55AM (#39325850)
"Crap docs aren't always the fault of the guys implementing though, sometimes there are time constraints that prevent proper docs being written."

I can always guarantee a stream of consciousness one note that should have most of the important data, and a few docs about the most important parts. It's up to management if they want me to spend time turning that one note into actual robust documentation that is easily read.

Re: by thelastparadise on Saturday February 10, 2024 @07:36AM (#39325731)
> the first thing I do is update / actually write the docs to make them usable.

OK so the docs are in sync for a single point of time when you finish. Plus you get to have the context in your head (bus factor of 1, job security for you, bad for the org.)

How about if we just write clean infra configs/code, stick to well known systems like docker, ansible, k8s, etc.

Then we can make this infra code available to an on prem LLM and ask it questions as needed without it drifting out of sync overtime as your docs surely will.

Wrong documentation is worse than no documentation.

Re: by ZoomerCretin on Saturday February 10, 2024 @02:54PM (#39329610)
Documentation? What for? It's self-documenting (to me, because I wrote it)!
Re: by jumploops on Saturday February 10, 2024 @11:25PM (#39332637)
Just to be clear, after I (and a few others left), they moved everything entirely to the cloud.

Even with documentation on the hybrid setup, they'd need to get a new on-prem environment up and running (find a colo, buy machines, set up the network, blah blah).

Re: by nyc_data_geek on Saturday February 10, 2024 @10:30AM (#39326820)
Yes, I do believe autoscaling is actually a good use case for public cloud. If you have bursty load that requires a lot of resources at peak which would sit idle most of the time, probably doesn't make sense to own what you need for those peaks.
Re: by throwawaaarrgh on Saturday February 10, 2024 @04:33AM (#39324854)
It's not an either/or. Many business both own and rent things.

If price is the only factor, your business model (or executives' decision-making) is questionable. Buy only the cheapest shit, spend your time building your own office chair rather than talking to a customer, you aren't making a premium product, and that means you're not differentiated.

Re: by Ir0nMan on Monday February 12, 2024 @02:27PM (#39349149)
Yep. This.
Re: by Scubabear68 on Friday February 09, 2024 @10:20PM (#39323251)
Elsewhere today I recommended RDS, but was thinking of small startup cases that may lack infrastructure chops.

But you are totally right it can be expensive. I worked with a startup that had some inefficient queries, normally it would matter, but with RDS it cost $3,000 a month for a tiny user base and not that much data (millions of rows at most).

Re: by rswail on Saturday February 10, 2024 @02:30AM (#39324330)
That sounds like the app needs some serious surgery.
Re: by j16sdiz on Saturday February 10, 2024 @12:30AM (#39323866)
In another section , they mentioned they don't have DBA, no app team own the database and the infra team is overwhelmed.

RDS make perfect sense for them

Re: by osigurdson on Saturday February 10, 2024 @12:49AM (#39323943)
Cloud was supposed to be a commodity. Instead it is priced like at burger at the ski hill.
Re: by dfgdfg34545456 on Sunday February 11, 2024 @04:25AM (#39333722)
If it is such a golden goose, then there will be other competitors come in and compete the price down.
Re: by dijit on Sunday February 11, 2024 @08:39AM (#39334901)
not really, the API lock-in and egregious egress fees will keep competitors at the door.

That: and trust is hard earned over a long tail which is harder if you are trying to compete on price.

Re: by osigurdson on Sunday February 11, 2024 @09:29AM (#39335195)
I think trust is the biggest factor. If you willingly lock yourself into a vendor specific product, that is obviously your own choice.
Re: by Sparkyte on Saturday February 10, 2024 @04:42AM (#39324896)
Data isn't cheap never was. Paying the licensing fees on top make it more expensive. It really depends on the circunstance a managed database usually has exended support from the compaany providing it. You have to weigh a team's expertise to manage a solution on your own and ensure you spent ample time making it resilient. Other half is the cost of upgrading hardware sometimes it is better to just outright pay a cloud provider if you business does not have enough income to outright buy hardware.There is always an upfront cost.

Small databases or test environment databases you can also leverage kubernetes to host an operator for that tiny DB. When it comes to serious data and it needs a beeline recovery strategy RDS.

Really it should be a mix self hosted for things you aren't afraid to break. Hosted for the things you put at high risk.

Re: by silisili on Saturday February 10, 2024 @05:14AM (#39325030)
Even for small workloads it's a difficult choice. I ran a small but vital db, and RDS was costing us like 60 bucks a month per env. That's 240/month/app.

DynamoDB as a replacement, pay per request, was essentially free.

I found Dynamo foreign and rather ugly to code for initially, but am happy with the performance and especially price at the end.

Re: by dfgdfg34545456 on Saturday February 10, 2024 @06:09AM (#39325281)
For big companies such as banks this cost comparison is not as straight forward. They have whole data centres just sitting there for disaster recovery. They periodically do switchovers to test DR. All of this expense goes away when they migrate to cloud.
Re: by graemep on Saturday February 10, 2024 @07:22AM (#39325644)
> All of this expense goes away when they migrate to cloud.

They need to replicate everything in multiple availability zones, which is going to be more expensive than replicating data centres.

They still need to test their cloud infrastracuture works.

Re: by nightfly on Saturday February 10, 2024 @06:28AM (#39325378)
> All of this expense goes away when they migrate to cloud.

Just to pay someone else enough money to provide the same service and make a profit while do it

Re: by dfgdfg34545456 on Saturday February 10, 2024 @07:25AM (#39325671)
Well corporations pay printers to do their printing because they don't want to be in the business of printing. It's the same with infrastructure, a lot of corporations simply don't want to be in the data centre business.
Re: by jabradoodle on Saturday February 10, 2024 @08:31AM (#39325997)
That's how nearly every aspect of every business works; would you you start a bakery by learning construction and building it yourself?
Re: by AtlasBarfed on Saturday February 10, 2024 @05:55PM (#39330994)
Construction is a one time cost. It infrastructure is in constant use.

It's like accounting and finance. Yeah a lot of companies use tax firms, but they all have finance and accounting in-house.

Re: by jabradoodle on Monday February 12, 2024 @07:19AM (#39344014)
I don't think that's the important distinction, lots of ongoing costs are outsourced.

Amazon may have their own shipping fleet, most retailers of smaller scale pay someone else to do it for profit.

Re: by AtNightWeCode on Saturday February 10, 2024 @06:27AM (#39325361)
In your case it sounds more viable to move to VMs instead of RDS, which some cloud providers also recommend.
Re: by prisenco on Saturday February 10, 2024 @07:42AM (#39325761)
From what I’ve read, a common model for mmorpg companies is to use on-prem or colocated as their primary and then provision a cloud service for backup or overage.

Seems like a solid cost effective approach for when a company reaches a certain scale.

Re: by hardolaf on Saturday February 10, 2024 @08:01AM (#39325876)
Lots of companies, like Grinding Gear Games and Square Enix, just rent whole servers for a tiny fraction of the price compared to what the price gouging cloud providers would charge for the same resources. They get the best of both worlds. They can scale up their infrastructure in hours or even minutes and they can move to any other commodity hardware in any other datacenter at the drop of a hat if they get screwed on pricing. Migrating from one server provider (such as IBM) to another (such as Hetzner) can take an experienced team 1-2 weeks at most. Given that pricing updates are usually given 1-3 quarters ahead at a minimum, they have massive leverage over their providers because they an so easily switch. Meanwhile, if AWS decides to jack up their prices, well you're pretty much screwed in the short-term if you designed around their cloud services.
Re: by fulafel on Saturday February 10, 2024 @09:28AM (#39326354)
I'd add another criticism to the whole quote:

> Data is the most critical part of your infrastructure. You lose your network: that’s downtime. You lose your data: that’s a company ending event. The markup cost of using RDS (or any managed database) is worth it.

You need well-run, regularly tested, air gapped or otherwise immutable backups of your DB (and other critical biz data). Even if RDS was perfect, it still doesn't protect you from the things that backups protect you from.

After you have backups, the idea of paying enormous amounts for RDS in order to keep your company from ending is more far fetched.

Re: by afpx on Saturday February 10, 2024 @10:17AM (#39326705)
That's the cost of two people.
Re: by raffraffraff on Saturday February 10, 2024 @02:10PM (#39329251)
I agree that RDS is stupidly expensive and not worth it provided that the company actually hires at least 2x full-time database owners who monitor, configure, scale and back up databases. Most startups will just save the money and let developers "own" their own databases or "be responsible for" uptime and backups.
Re: by rr808 on Saturday February 10, 2024 @02:20PM (#39329344)
For a couple hundred grand you can get a team of 20 fully trained people working full time in most parts of the world.
Re: by illumin8 on Saturday February 10, 2024 @09:01PM (#39332035)
This is because you are using SQL Server. Microsoft has intentionally made cloud pricing for SQL server prohibitively expensive for non-Azure cloud workloads by requiring per-core licensing that is extremely punitive for the way EC2 and RDS is architected. This has the effect of making RDS vastly more expensive than running the same workload on bare metal or Azure.

Frankly, this is anti-competitive, and the FTC should look into it, however, Microsoft has been anti-competitive and customer hostile for decades, so if you're still using their products, you must have accepted the abuse already.

Re: by pconstantine on Monday February 12, 2024 @02:46PM (#39349427)
Totally agree. It's cherry-picking some weird case that's not even close to a typical for startup.
Re: by WuxiFingerHold on Saturday February 10, 2024 @11:22PM (#39332624)
Also, it is often overlooked that you still need skilled people to run RDS. It's certainly not "2-clicks and forget" and "you don't need to pay anyone running your DB".

I haven't run a Postgres instance with proper backup and restore, but it doesn't seem like rocket science using barman or pgbackrest.

Re: by parentheses on Sunday February 11, 2024 @03:38AM (#39333481)
While I agree that RDS is expensive, you're making two false claims here:

1. Hiring someone full time to work on the database means migrating off RDS

2. Database work is only about spend reduction

Re: by pconstantine on Monday February 12, 2024 @02:44PM (#39349399)
People, who use MSFT SQL server in 2024 should suffer. For everybody else there's always Neon.