News for hackers. Stuff that matters.
Re:Cutting down AWS cost by $150k per year simply by shuttin... by darth_avocado on Monday January 22, 2024 @12:02PM (#39091890)
It is unfortunate that cost management isn’t something most engineers keep an eye out for on a regular basis. Spinning up unnecessary resources, not cleaning up resources properly once not needed, writing inefficient code, etc. all quickly adds up to hundreds of thousands of dollars per month in big companies.

I once found a “test” db cluster from an engineer who hadn’t worked in the company for 3 years. We were paying 300k yearly for it before discounts. It took me a literal click to shut it down. And I’m not proud of it but, had to send out an org wide email on the savings achieved (corporate politics :shrug:).

Re:Cutting down AWS cost by $150k per year simply by shuttin... by pjc50 on Monday January 22, 2024 @12:07PM (#39091969)
The huge achievement of Amazon was designing a system and selling it to people where developers no longer had to pre-approve spending. Previously developers were hamstrung by purchase order requirements; it could take weeks to authorize a single computer. Now the pendulum has swung in the direction. Developers can spend unlimited amounts of company money without realizing, billed in arrears.

And in many cases this is a huge net win! After all, there's another way to waste company money invisibly: design a process which requires meetings and waiting while work is held up.

Re:Cutting down AWS cost by $150k per year simply by shuttin... by pyeri on Monday January 22, 2024 @12:29PM (#39092322)
That's the whole ploy with Agile, isn't it? In the classical SDLC or Waterfall paradigm, everything was pre-approved and signed off, not just the cost or billing but even the software design itself. Any change in the process and the designers had to raise a change request. Agile changed all of that and now we know how bad things can get with that.
Re:Cutting down AWS cost by $150k per year simply by shuttin... by tbalsam on Monday January 22, 2024 @12:36PM (#39092454)
No, Agile is about tight development loops. When weaponized/used by large corporations it often times turns into a Stay-Puft man of sorts, but as someone who _hates_ processes normally, I actually kinda like it and Kanban when done well.

It basically helps keep things clean with decomposition and doesn't necessarily hamstring older devs as much while giving a good guideline for younger devs to work in. All things considered, it seems like not a bad system to me, and the team customizing the process to their own needs is nice as well.

There's a million ways for it to go wrong, but it's not too terrible on the whole I thinks. <3 :"))))

Re:Cutting down AWS cost by $150k per year simply by shuttin... by ska on Monday January 22, 2024 @01:15PM (#39093058)
> who _hates_ processes normally,

There is no such thing as "no process"; is something you always have whether you talk about it or not. The often heard "I hate process" is counterfactual then - what it really means is "I hate process that I see as intrusive/wasteful/whatever".

The +'ves you are listing are what comes from looking at how things are actually done and doing some of it a bit more thoughtfully.

The common -ves often come from stakeholders outside of developement injecting their needs ... sometimes this is unavoidable (e.g. regulatory) sometimes it is just political, but either way there are better and worse ways to do it.

Re:Cutting down AWS cost by $150k per year simply by shuttin... by pjc50 on Tuesday January 23, 2024 @04:28AM (#39101283)
But see https://news.ycombinator.com/item?id=39092563 ; the cost of pre-approval can be pretty large, as is the cost of change once you discover that you've frozen in the wrong thing. Agile benefits both consulting situations where the client genuinely doesn't understand their own needs, and also startups where you're building a new product and need to rapidly iterate in response to market feedback.

(Escalating costs of pre-approval, and the need to design around every possible objection, are a big part of why physical infrastructure costs so much more in the West!)

Re:Cutting down AWS cost by $150k per year simply by shuttin... by throwup238 on Monday January 22, 2024 @12:10PM (#39092011)
We should be getting kickbacks from Amazon for all the work we’ve done for their bottom line.
Re:Cutting down AWS cost by $150k per year simply by shuttin... by marvin on Monday January 22, 2024 @12:51PM (#39092707)
We kinda do; Amazon puts upwards pressure on US engineering salaries.
Re:Cutting down AWS cost by $150k per year simply by shuttin... by LunaSea on Monday January 22, 2024 @02:33PM (#39094225)
What? No? They are behind all major tech companies and notoriously one of the worst employers too.
Re:Cutting down AWS cost by $150k per year simply by shuttin... by teaearlgraycold on Monday January 22, 2024 @01:51PM (#39093615)
I assume all of the unused resources end up subsidizing the rest of them.
Re:Cutting down AWS cost by $150k per year simply by shuttin... by amichal on Monday January 22, 2024 @12:43PM (#39092563)
Overall good points but don't forget that pre-approval processes resulted in asking for resources that exceeded the near term needs and once approved ongoing costs were rarely fully reviewed. I have personal experience with "enterprise" clients making a huge months long process to get server resources, reminding us that changes would take 30+ days. when the project was over and we did everything we could to let them know that the servers could be spun down or put to other uses we got back a "ok thanks!" only to find them still running our project code YEARS later. This is infra that was costing them about 1 engineer FTE per year, not even a 10$/mo toy env
Re:Cutting down AWS cost by $150k per year simply by shuttin... by mytailorisrich on Monday January 22, 2024 @01:04PM (#39092926)
Yes but it is up to a company to control its spending. It must have a process and policy in place to deal with this. It's not Amazon's fault if it hasn't.
Re:Cutting down AWS cost by $150k per year simply by shuttin... by cduzz on Monday January 22, 2024 @12:20PM (#39092179)
I think "engineer" isn't really the correct word to use for the artisans who build much of the tooling used by most companies.

An engineer either wears a striped hat and drives a train, or, went to a credentialed school and passed a bunch of test and is allowed to sign documents that state "this thing, if built this way, won't collapse and kill people."

It is expected that an engineer can predict with reasonable accuracy the expense and timeline of a project, and how to maintain the resulting thing, without resorting to voodoo like "scrum velocity." In large part that's because engineers stick to doing things that are well understood and predictable, and if there's risk they resolve the risk before undertaking the project. (Is there bedrock over here upon which to build a foundation? I don't know; let's find out first!). Sure, there are engineering disasters even today -- buildings that unexpectedly lean over and door/wall things that unexpectedly fly off the side of airplanes, but those are typically organizational / process problems not "engineering doesn't work" problems.

Re:Cutting down AWS cost by $150k per year simply by shuttin... by reactordev on Monday January 22, 2024 @12:33PM (#39092410)
“engineers stick to doing things that are well understood and predictable”

I’m calling BS on this. If this were true, we’d still be a ground species. Engineering has been and will always be about creating something electrical, mechanical, computerized, or all, that solves a problem. Understood or not. Engineers are not oracles. They can not predict whether a tower built in Italy will eventually begin to lean due to erosion. They can not predict that a steel beam rated for 300T of force would break at 180T. They can not predict a rogue developer removing a package from underneath their dependency tree.

You can give estimates all you want but you are still guessing.

If engineers were as you say they are, we would never have delays, we would never have traffic jams, we would never have crap software, we would never have flight.

Re:Cutting down AWS cost by $150k per year simply by shuttin... by practicemaths on Monday January 22, 2024 @12:47PM (#39092638)
“Engineering is the art of modelling materials we do not wholly understand, into shapes we cannot precisely analyse so as to withstand forces we cannot properly assess, in such a way that the public has no reason to suspect the extent of our ignorance.” - Dr. AR Dykes
Re:Cutting down AWS cost by $150k per year simply by shuttin... by reactordev on Monday January 22, 2024 @12:50PM (#39092698)
Ah the eloquence of Dr. AR Dykes, perfectly said. Thank you.
Re:Cutting down AWS cost by $150k per year simply by shuttin... by noboostforyou on Monday January 22, 2024 @03:01PM (#39094613)
I am partial to the following one about computers:

"A cpu is literally a rock that we tricked into thinking."

Re:Cutting down AWS cost by $150k per year simply by shuttin... by cduzz on Monday January 22, 2024 @12:54PM (#39092755)
Engineers manage risk and cost. They certainly make mistakes, like those couple buildings that are famously leaning over in SF and NYC, or the citycorp center where they got the wind sheer loads wrong and had to hot patch the building.

But looking at the malarkey that goes on in "software engineering" or whatever -- clearly not engineering, at least not where I've seen it.

Engineering: a process of repeatably solving an understood problem predictably.

Craft: a process of solving an understood problem.

Science: a process of solving a problem without an exactly understood outcome.

Art: a process of working.

These are all made-up definitions.

I'd expect a software engineer to give me a system that locally caches and verifies distribution artifacts and validates changes -- a craftsperson who gives me a tool chain that yeets goo from the internet and builds on that without validation is not, in fact, an engineer. They could be quite practiced at the art of building working systems, but they're not managing risk....

Re:Cutting down AWS cost by $150k per year simply by shuttin... by dehrmann on Monday January 22, 2024 @01:01PM (#39092880)
What makes software engineering special is the systems are more complex and are cheaper to test and break. You get a completely different engineering culture when you can roll back a bad change after seeing it fail during the canary push. That, and what's usually on the line is money, not life. I'd feel a lot better making a $1M mistake than making a mistake that killed someone.
Re:Cutting down AWS cost by $150k per year simply by shuttin... by ExoticPearTree on Monday January 22, 2024 @01:16PM (#39093076)
> Engineering: a process of repeatably solving an understood problem predictably.

We call it help desk, not engineering.

Re:Cutting down AWS cost by $150k per year simply by shuttin... by gottorf on Monday January 22, 2024 @02:37PM (#39094276)
> Engineers manage risk and cost.

"Any idiot can build a bridge that stands, but it takes an engineer to build a bridge that barely stands."

Re:Cutting down AWS cost by $150k per year simply by shuttin... by robocat on Monday January 22, 2024 @02:47PM (#39094410)
> An engineer [] went to a credentialed school and passed a bunch of test and is allowed to sign documents that state "this thing, if built this way, won't collapse and kill people."

Ahhh - that old craptacular definition. You completely ignore mechanical engineers, chemical engineers, electrical & electronics engineers. Not all engineers make bridges.

Secondly, the implied cause and effect even within civil engineering is a fantasy. Signatures on documents by credentialed engineers doesn't prevent disasters as you noted: Bridges fall down, buildings burn. Read the engineering reports on civil engineering disasters, and look at the consequences for the engineers involved.

You do some handwaving about organizational/process problems, but actually that is the key to safe engineering. Organisations deliver engineering projects and they do it across jurisdictional borders using insurance and liability and with a variety of other means that work: "signatures don't prevent disasters".

Lockheed Martin's skunk-works and SpaceX are real engineering. Any good definition of engineering needs to encompass an extremely wide variety of activities.

Engineering is compromise. I have no love for Musk but him saying build that actuator for less than $5k is actually true engineering: https://news.ycombinator.com/item?id=39085892

I would like to know the psychology behind why people wish to believe credentialed signatures are so powerful? Maybe a cross between two concepts #1: "that individual engineers run the world" and #2: "that retributive punishment of individuals works as a deterrent". I think concept #1 comes from the egotist idea of most engineer-types that we are the center of everything (I need a whole article to explain the concept). I think concept #2 is related to beliefs about the value of incarceration and also punishment beliefs derived from religion (especially in the USA where prisons are not fixing problems?).

Edit: issue #3: the idea that we should make rules about what words mean. It takes a certain worldview to think words should be defined rather than evolve (or worse that words should be part of a justice system)

Re:Cutting down AWS cost by $150k per year simply by shuttin... by cduzz on Monday January 22, 2024 @03:34PM (#39095039)
> Ahhh - that old craptacular definition. You completely ignore mechanical engineers, chemical engineers, electrical & electronics engineers. Not all engineers make bridges.

I suppose you've got an engineering degree in pedantic engineering? Engineers manage cost and risk. The skunk-works stuff is marginally "science" not "engineering" given the relatively large budgets and relative lack of "we know this works." Cern is similarly an enormous engineering enterprise in that it's a huge stack of "we know this works" in service of "we're not sure what this will do"

A discussion of how "software engineers" deliver projects with neither cost or risk as part of the process implies, to me, that they're not engineers.

Re:Cutting down AWS cost by $150k per year simply by shuttin... by robocat on Monday January 22, 2024 @05:58PM (#39096753)
You are the one trying to push your definition of engineering.

I provided counter-examples that show engineering encompasses a lot more than your definition.

I simply don't understand why anyone thinks writing software is somehow uniquely not "real" engineering. Somehow we are indoctrinated to believe that it isn't but all the evidence seems to show software engineering is a valid description.

I have no lack of experience watching the fuck-ups made by electronics engineers, or the fuck-ups made by mechanical engineers. You appear to want to define engineering only as certified civil engineering. And I've seen enough of their fuck-ups too, with signatures. In fact I'll ask my bridge engineer friend from uni about it! Unfortunately my bridge building grandad is dead so I can't ask him.

Re:Cutting down AWS cost by $150k per year simply by shuttin... by cduzz on Monday January 22, 2024 @07:26PM (#39097617)
The vast majority of people I encounter with some "engineering" title, in software (or the related "Architect") are in fact not trained as engineers or architects, in any field.

A site reliability "engineer" or a software "engineer" is not an actual engineer just because they've got that in their title or job description. If I were to hire a "chemical engineer" position and instead hired a chemist, or a mechanic, or a rando who's cooked meth, I may end up with things working okay, or I might end up with a serious mess, even if those people I hire call themselves "engineers" (but in fact have no formal training as such).

I'm not sure to what degree credentials matter, but do credentials matter more than "not a god damned bit" ?

I'm not saying the title makes you "not an idiot" -- people gonna people -- but attention to "cost" and "risk" is (theoretically) one of the distinguishing characteristics of engineering training vs ... "mather" or "programmer" or "philosopher".

Re:Cutting down AWS cost by $150k per year simply by shuttin... by robocat on Monday January 22, 2024 @08:25PM (#39098233)
Yeah, the debasement of meaning is annoying - vice-president is one I hate. Another one that surprises me from the US is "licensed nail technician".

I have a bachelor of engineering title I can use with my name, but that is another distinct type of bullshit.

In New Zealand one relevant legal certification is CPEng which you can apply for after receiving your degree and working for a few years: https://www.engineeringnz.org/join-us/cpeng/ And apparently our government agreed in 2022 to introduce a new licensing regime for engineers doing safety-critical work.

But in an international world, how relevant are certified individuals? When I purchase a stove from a US brand and it catches fire, there needs to be other liability/retribution/corrective systems to deal with the problem. It matters little to me who signed off on the product in the US.

Can I import custom structural steel beams? How many New Zealanders have signed off on this steel construction: https://ccc.govt.nz/the-council/future-projects/major-facili... We need a new stadium because the last one broke. Unfortunately it wasn't insured due to some cockup at the city council (which I suspect had zero retribution on the people that cocked up - I wonder if they signed bits of paper?).

Over-credentialisation is a problem too - where is the right balance? The shift to everyone needing credentials is fucked. My friends (nurses, teachers) literally weep at the absolute trash they have to "learn" for their credential. I also vividly remember the crap I needed to disgorge to get my degree.

I don't know what the answer is, but I honestly believe most credentials are pointless waste and adding more credentials is not actually effective. Neither do I believe that that the anarchy of libertarian free markets are a workable answer.

Re:Cutting down AWS cost by $150k per year simply by shuttin... by happymellon on Monday January 22, 2024 @12:22PM (#39092210)
I found that the problem happens mostly when companies

1. Don't ask developers how much something costs, engineers love optimisation, getting as much as possible out of a system for cheap is great fun.

2. Lock down the UI, so devs can't even find out how much things cost. That's my current situation. Why block the billing dashboard, then expose it through billing dashboard tools that are not really any better, and in many ways worse?

It's rhetorical really as I know why. Terrible architecture from "enterprise". Stick everything in a single account so it's hard to figure out how much is your spend. All 3000 databases, and make sure your k8s cluster is 5 8XL boxes so no one can scale down excess capacity.

Classic I Burn Money consultancy!

Re:Cutting down AWS cost by $150k per year simply by shuttin... by tuananh on Monday January 22, 2024 @12:26PM (#39092276)
> 2. Lock down the UI, so devs can't even find out how much things cost. That's my current situation. Why block the billing dashboard, then expose it through billing dashboard tools that are not really any better, and in many ways worse?

This is so true. billing transparency is very important.

in the past, i had a case like this: dev accidentally enable backup policy for test database with no retention. finops think that db backup is important and ignore it. dev has no access to billing and have no idea what's creeping up the bill

Re:Cutting down AWS cost by $150k per year simply by shuttin... by darth_avocado on Monday January 22, 2024 @12:28PM (#39092317)
Locking down the UI is definitely a problem. Another problem I have seen is, not being able to accurately tell even if you have the UI.
Re:Cutting down AWS cost by $150k per year simply by shuttin... by happymellon on Monday January 22, 2024 @12:31PM (#39092356)
We had a battle to get AWS console access in the first place, after that I had to deal with:

> You need to request access every 60 days.

Luckily it's now added to my permanent role, but even then no billing access? FFS.

Re:Cutting down AWS cost by $150k per year simply by shuttin... by macNchz on Monday January 22, 2024 @01:01PM (#39092883)
> getting as much as possible out of a system for cheap is great fun

In certain circumstances, absolutely, however it's extremely aggravating to be in the position of being constantly pestered to ship features faster without the authority to overprovision some of the infrastructure the software runs on.

Waking up in the middle of the night because we saved money by allocating too little disk for the primary database or because the latest release included new dependencies that increased memory usage and the OOMKiller is picking off web servers like a wolf in the lamb's pen, or we're just swapping our way to hell while web requests 502...eh. Not for me.

More visibility into costs, though, absolutely agreed. Engineers should know that when they turn on some new cool serverless gizmo and then forget about it, it's costing $ each month.

Re:Cutting down AWS cost by $150k per year simply by shuttin... by happymellon on Monday January 22, 2024 @01:22PM (#39093150)
Completely agree.

I didn't mean that engineers love having no control over their systems, I just see the labours of love that get posted here about getting nginx to throw out 1000 pages a second on an Atari 800, or getting LLMs designed for $2000 GPUs running on a phone.

The question should be, we currently cost $X a month and we need to half it because [reason], what can we do to bring it down? Which might be reducing hardware, or maybe something else, might be both. Puzzles can be fun.

Re:Cutting down AWS cost by $150k per year simply by shuttin... by malfist on Monday January 22, 2024 @12:33PM (#39092404)
Lots of that is just bad design by going the easiest route.

It's easy not to grant engineers access to the billing dashboard.

It's easy to put everything in the same aws account.

Inside Amazon, we're supposed to set up new aws accounts for every service and realm, so we know how much X service's beta environment is costing

Re:Cutting down AWS cost by $150k per year simply by shuttin... by happymellon on Monday January 22, 2024 @01:16PM (#39093085)
Indeed, you need to share resources?

Plenty of ways of doing that, like making a cross account shared VPC for example.

Everything is still accountable.

Re:Cutting down AWS cost by $150k per year simply by shuttin... by righthand on Monday January 22, 2024 @12:25PM (#39092253)
I had to implement a 2nd deploy for a QA environment, and my first question to the infrastructure team was “won’t this be costly is there a better way to handle this?” They shrugged off the cost and said they would optimize my deploy once I was done with the initial implementation. 6 months later their optimization was to undo all the work not because it wasn’t a good implementation but because it revealed how much non-optimization had went into the QA environment before I even touched it. A lot of cost is probably due to the “we just taped these two things together” strategy for lower environments.
Re:Cutting down AWS cost by $150k per year simply by shuttin... by dilyevsky on Monday January 22, 2024 @12:27PM (#39092291)
> It is unfortunate that cost management isn’t something most engineers keep an eye out for on a regular basis.

That’s because they were explicitly told not to worry about costs for the last 10 years so majority of ICs at this point never had to do it their entire careers

Re:Cutting down AWS cost by $150k per year simply by shuttin... by racl101 on Monday January 22, 2024 @12:25PM (#39092258)
I tend to have the opposite problem. I obsesses over the cost of things, and am pretty bashful about bringing it up to my manager, and he's always surprised that scaling some resources doesn't cost more. But I learned the hard way as a contractor about letting these resources run crazy and had to pay out of pocket so I have PTSD about it, which is why I'm vigilant.
Re:Cutting down AWS cost by $150k per year simply by shuttin... by lnxg33k1 on Monday January 22, 2024 @12:07PM (#39091964)
I get paid the same whether or not I spend time going around to save company cost, it’s not like they’re going to share their savings, then shareholders get their juice, management gets their juice and Im the clown who went out of my way for them, who cares I do software not cost management
Re:Cutting down AWS cost by $150k per year simply by shuttin... by pixl97 on Monday January 22, 2024 @12:09PM (#39091994)
Which is why your slice of the organization gets a cloud budget. Don't keep your budget under control, well no bonuses for the employees.

>Show me the incentive and I will show you the outcome

Re:Cutting down AWS cost by $150k per year simply by shuttin... by wizerdrobe on Monday January 22, 2024 @12:14PM (#39092063)
Jokes on corporate, I don’t have ISO, RSU, or a bonus and my raises are always below inflation.
Re:Cutting down AWS cost by $150k per year simply by shuttin... by lnxg33k1 on Monday January 22, 2024 @12:16PM (#39092098)
Oh for the inflation thats awesome, you know how many interviews with engineers Ive had saying they were at a place for 10 years, but the company had a raise cap of 2%, then saying that they wouldn’t hire me because I couldn’t give assurances to be a likeminded clown making corporate rich
Re:Cutting down AWS cost by $150k per year simply by shuttin... by ponector on Monday January 22, 2024 @12:15PM (#39092082)
But bonuses are not related to the cloud costs.

Nobody in engineering cares about spending because there is no benefits on doing so.

Even more: most people are on fixed salary and will get paycheck anyway no matter how low their effort is.

Re:Cutting down AWS cost by $150k per year simply by shuttin... by pixl97 on Monday January 22, 2024 @02:25PM (#39094116)
Hence

>>Show me the incentive and I will show you the outcome

Re:Cutting down AWS cost by $150k per year simply by shuttin... by lnxg33k1 on Monday January 22, 2024 @12:11PM (#39092023)
>Show me the incentive and I will show you the outcome

Would print this phrase on every angle of the offices

Re:Cutting down AWS cost by $150k per year simply by shuttin... by freedom-fries on Monday January 22, 2024 @02:16PM (#39094007)
Look at Mr bonuse-pants here. Y'all get bonus for writing software?
Re:Cutting down AWS cost by $150k per year simply by shuttin... by krageon on Tuesday January 23, 2024 @04:16AM (#39101186)
This is fun, but these policies are based on the fact that your managers don't care about you and in fact prefer you remain as poor as they can legally make you. So any kind of reasoning along these lines ("well this makes sense") just is not real and won't ever become real.
Re:Cutting down AWS cost by $150k per year simply by shuttin... by tuananh on Monday January 22, 2024 @12:04PM (#39091909)
that's one of the main reason we made this feature "opt-out".

if you want to keep it up, you have to tag it.

once you tag it, it can opt-out 7 days. then you have to extend it (simply chat with our bot)

Re:Cutting down AWS cost by $150k per year simply by shuttin... by zikduruqe on Monday January 22, 2024 @12:05PM (#39091933)
https://cloudcustodian.io

Create a rule and shame them on Slack.

Re:Cutting down AWS cost by $150k per year simply by shuttin... by onyxringer on Monday January 22, 2024 @12:13PM (#39092052)
Don't you know that "Developer's time is expensive"? :D
Re:Cutting down AWS cost by $150k per year simply by shuttin... by dangus on Monday January 22, 2024 @12:07PM (#39091960)
Well, it’s definitely fathomable. Does my employer have cost control baked into their proceses, tooling, and culture or are they rushing me to get projects out the door leaving barely enough time to make sure they’re production-ready?

Most places I’ve worked had no formal production readiness review before launching infrastructure.

Re:Cutting down AWS cost by $150k per year simply by shuttin... by znpy on Monday January 22, 2024 @12:57PM (#39092801)
This is entirely artificial: I now work at a company where we know very clearly what our infrastructure costs. Yes, we know the exact costs (what was negotiated, not what is on the public pages).

And we celebrate costs slashing as much feature delivery and other stuff.

But this is entirely a management problem: at my previous job, only one manager (skip-level manager from my point of view) knew what exactly were we paying for infrastructure.

That moron wouldn't share that information with us engineers managing infrastructure of course, so there were a lot of infrastructure choices that didn't really made sense according to the public prices but (I guess?) made sense according to a price sheet we didn't know.

So we didn't know what we were spending, didn't have the basic data to estimate the price of a new solution or a new service and didn't have the data to determine how much would we be saving by making changes (optimizing stuff etc).

I fought that battle for a bit but then i just said "GFYS, i'm not going to have fights with you so that you can save money" and let go. Later i left the company completely.

Former colleagues tell me it's even worse now: there are consultants from the cloud provider involved, they know the pricing deals, and whenever the topic comes up the manager shushes the consultant so that the engineers don't hear the prices.

tl;dr: it's an entirely artificial problem, and it's most likely a cultural/management problem.

edit: and i'm not even talking about incentives, as somebody else has correctly pointed out.

Re:Cutting down AWS cost by $150k per year simply by shuttin... by VoodooJuJu on Monday January 22, 2024 @01:10PM (#39093007)
And that's how insignificant the costs of cloud providers are in the grand scheme of things. It's a lot of money to a bootstrapping startup, but for the vast majority of these cloud providers' customers, it's a rounding error that's easily forgotten for 3 years.

And that's precisely why you and your little bootstrapper or indie firm should not be using globocloud: you do not have mountains of cash to piss away. Bare metal is trending again. And in this downturn, it's no wonder why. Smaller companies are getting smarter and more efficient. They've decided to chase money instead of cargo cults.

Globocorps burning cash on globocloud is not a signal for small fish to do the same - it's a signal to do the polar opposite. You're not going to become like them by copying what they're doing now. It will not work for you. Globocloud isn't successful because they shovel cash into AWS's shredder, they shovel cash into the shredder because they're successful.

Re:Cutting down AWS cost by $150k per year simply by shuttin... by hibikir on Monday January 22, 2024 @01:30PM (#39093295)
You'd be surprised. I've seen AWS bills well in the 9 figures. It's just that fixing expensive designs is, in itself, quite expensive, and many of those very large corps have hiring practices that don't allow them to complete for the top of the market. Sometimes there's tens, if not hundreds of millions in savings a year, but corporate sclerosis makes it very difficult for broad cost-saving initiatives to be identified and approved.

It's the same issue in any large organization: Large levels of success somewhere allow for large levels of waste somewhere else, but often the waste is not required for the success to exist: The success just makes the organization complacent.

Re:Cutting down AWS cost by $150k per year simply by shuttin... by mr_00ff00 on Monday January 22, 2024 @01:19PM (#39093102)
Would be a random coincidence, but if you work at a large bank, I think I may know the team that had that 300k test db lol.
Re:Cutting down AWS cost by $150k per year simply by shuttin... by sotix on Monday January 22, 2024 @01:19PM (#39093104)
I consistently tried to push for cost management at my last job, but the product manager just wanted to push new features he could show off to management above him. We let costs inflate to ridiculous levels despite my constant discussion around the topic. Software engineers ultimately had no say in the matter.

I was laid off this week in a mass layoff because the company doesn’t have enough money to pay all of us anymore. It’s disappointing to see, and I wonder how many other teams ignored these optimizations and how much unnecessary total cost it all summed to.

Re:Cutting down AWS cost by $150k per year simply by shuttin... by ExoticPearTree on Monday January 22, 2024 @01:24PM (#39093204)
In the organization that I work in, costs are transparent to everyone involved and most people are aware of the need to keep costs as low as possible.

One of the downsides with this approach is that engineers/developers are not very good business people and don't really understand the notion of "the cost of doing business". And from time to time we have issues with "but it costs $70 more per month", and spend $1000 to optimize those $70 :)

In the end, even with some of the wrinkles mention above it helps and saves money when costs are transparent and readily available for anyone.

Re:Cutting down AWS cost by $150k per year simply by shuttin... by nlawalker on Monday January 22, 2024 @01:45PM (#39093523)
This topic got a lot of discussion in "I accidentally saved my company half a million dollars": https://news.ycombinator.com/item?id=38069710
Re:Cutting down AWS cost by $150k per year simply by shuttin... by myaccountonhn on Monday January 22, 2024 @01:59PM (#39093742)
What I've observed is that people don't really keep track of what they are spending. I like to set up weekly newsletters that show costs and also if there has been a decrease or increase. In bigger corps, you also should have team based tagging of resources so that specific teams get exactly what they are spending. At the very least, managers will look each week and be like "why did costs increase this week? What's going on?" even if the engineers don't care. "What's get measured gets managed" as they say.