News for hackers. Stuff that matters.
Re:Switching to AWS Graviton slashed our infrastructure bill... by c7DJTLrn on Wednesday November 30, 2022 @03:17PM (#33807017)
Cloud cost optimisation is underrated. In the companies I've worked in nobody has really given a shit (at least not under normal economic circumstances). In the industry there's a strong avoidance of ARM compute instances for no good reason. If I were building from scratch today I would definitely go with Graviton.
Re:Switching to AWS Graviton slashed our infrastructure bill... by pclmulqdq on Wednesday November 30, 2022 @03:32PM (#33807215)
I think the main reason is "I want to run the same binaries locally that I run in the cloud," and it's a pretty valid one. However, it's also an expensive one sometimes.
Re:Switching to AWS Graviton slashed our infrastructure bill... by jacobwg on Wednesday November 30, 2022 @03:36PM (#33807272)
Anecdotally, this is starting to shift with M1 MacBooks, Graviton is looking more attractive for precisely that architecture parity reason for teams using majority M1 devices.
Re:Switching to AWS Graviton slashed our infrastructure bill... by gryf on Wednesday November 30, 2022 @03:59PM (#33807594)
Yeah if only. Our ops people are too uneducated to be able to deploy anything Apple. Literally there are armies of factory pressed Windows monkeys but nothing in the Apple space.

Note to apple: please start concentrating on the enterprise sector. We're dying over here. My Dell weighs 3x my personal M1 MBP, has a shitty keyboard with keys designed for Borrowers, the battery lasts 8 minutes and it reduces my sperm count if I put it on my lap. It feels like I have a ball and chain around my ankle 24/7. My only escape is WSL2 which is broken as fuck as well (can't run services, cron jobs, X problems etc) and we can't install a simple non-WSL VM on the node because Device Guard requires hyper-v to be enabled excluding sensible and pure VM options like VirtualBox. Docker for windows is a comedy of errors too.

Re:Switching to AWS Graviton slashed our infrastructure bill... by sbarre on Wednesday November 30, 2022 @04:29PM (#33808011)
For what it's worth, I work in a large 30,000 employee company. Everyone gets a Windows machine by default.

6 years ago our department of 200 people "went rogue" and started provisioning Macs, because it was the only way we could hire developers and provide a good developer experience for the work we were doing.

This took some convincing, but it was possible. We agreed to be unsupported by the in-house help desk, but we had 2 people in IT that supported us for provisioning and fixing machines, and sorting out a small amount of required enterprise software like a Cisco VPN client and some fleet management background agents.

Otherwise, we self-organized support over Slack and in-office and also made use of Apple's business support directly.

As of earlier this year, our department is now over 600 people, and we've given our internal IT enough incentive to officially support Macs, which they now do, alongside Windows.

They use some kind of MDM software to manage and update and monitor our Macs the same as they do Windows.

There are also now additional much larger teams in the organization exploring Mac adoption where it makes sense for their developers too, and we could soon have thousands of Macs in use.

So it's definitely possible, even if you have to start small.

Re:Switching to AWS Graviton slashed our infrastructure bill... by gryf on Wednesday November 30, 2022 @04:33PM (#33808072)
Unfortunately we're in a regulated industry so getting a PO signed off for one Mac is nigh on impossible without involving the entire corporate machinery.
Re:Switching to AWS Graviton slashed our infrastructure bill... by sbarre on Wednesday November 30, 2022 @10:36PM (#33812033)
We are also a highly regulated industry. Not finance but pretty close.

I sympathize with you though, it sounds like there's no will to do it, which sucks :(

Re:Switching to AWS Graviton slashed our infrastructure bill... by lostmsu on Wednesday November 30, 2022 @04:44PM (#33808215)
Are your developers forced to use laptops?
Re:Switching to AWS Graviton slashed our infrastructure bill... by gryf on Wednesday November 30, 2022 @04:56PM (#33808385)
Unfortunately yes. I would rather a desktop but they don't know how to pay half as much for the same specification. The desks in the office are all equipped with docks and expensive WiFi mesh driven by COVID mentality so that is the status quo.

Just send me a fucking workstation. Nope too hard.

Re:Switching to AWS Graviton slashed our infrastructure bill... by scarface74 on Wednesday November 30, 2022 @05:09PM (#33808545)
Who wants to be in an office in 2022 unless necessary (specialized hardware, etc.)
Re:Switching to AWS Graviton slashed our infrastructure bill... by gryf on Wednesday November 30, 2022 @05:22PM (#33808730)
Exactly that. I don't and I won't do it again.
Re:Switching to AWS Graviton slashed our infrastructure bill... by scarface74 on Wednesday November 30, 2022 @04:49PM (#33808291)
Do you propose they carry desktops back and forth from their office and to conference rooms?
Re:Switching to AWS Graviton slashed our infrastructure bill... by lostmsu on Wednesday November 30, 2022 @05:14PM (#33808624)
Every conference room could have a device to facilitate presentation.
Re:Switching to AWS Graviton slashed our infrastructure bill... by scarface74 on Wednesday November 30, 2022 @08:17PM (#33810890)
What device is going to have my PowerPoint slides, my IDE, my logged in AWS account, etc and everything else I need for demos?

Where I work, every conference room in every corporate office has a TV and a dongle that you can plug in via either HDMI or USB C.

Re:Switching to AWS Graviton slashed our infrastructure bill... by lostmsu on Wednesday November 30, 2022 @10:58PM (#33812147)
Any thin client would do.
Re:Switching to AWS Graviton slashed our infrastructure bill... by scarface74 on Wednesday November 30, 2022 @11:49PM (#33812475)
I’m going to have a “thin client” with my exact setup and my IDE, PowerPoint slides etc? Are you proposing that I work from a thin client? What happens when I’m working from home?
Re:Switching to AWS Graviton slashed our infrastructure bill... by lostmsu on Thursday December 01, 2022 @12:30AM (#33812720)
It's only for meeting rooms.
Re:Switching to AWS Graviton slashed our infrastructure bill... by scarface74 on Thursday December 01, 2022 @12:37AM (#33812761)
So in a meeting, is this person who is presenting going to copy everything they need from their desktop? If it’s a group meeting will no one else have their computers with them?
Re:Switching to AWS Graviton slashed our infrastructure bill... by happymellon on Thursday December 01, 2022 @08:27AM (#33815915)
1. I don't think you understand what a thin client is.

2. I have found that face-to-face meetings where other people bring their laptops usually means that they get distracted and don't participate.

Re:Switching to AWS Graviton slashed our infrastructure bill... by scarface74 on Thursday December 01, 2022 @08:59AM (#33816257)
You’re right, I am new to this whole cloud stuff, it’s not like I actually work for the largest cloud provider in the world in the consulting department where it’s my job to know this (hint: I do).

It’s also not like I wrote my first line of code in 1986 (Hint guess what the 74 in my name signifies).

Now that hopefully we can dispense with the idea that I don’t know how this computer stuff works. Please answer the question. How do I get everything I need from my computer to this thin client? How do I keep my environment in sync with this thin client?

Re:Switching to AWS Graviton slashed our infrastructure bill... by sbarre on Thursday December 01, 2022 @09:29AM (#33816647)
My friend, you are being trolled.. Your responders keep moving the goal posts on you.

You should just walk away.

Re:Switching to AWS Graviton slashed our infrastructure bill... by sbarre on Wednesday November 30, 2022 @10:39PM (#33812050)
Our company issues laptops yes, because everyone can choose to either work at home or in the office.

That said, we also supply external monitors, keyboards and mice for anyone who wants one, and every worker has a discretionary budget to spend on home office accommodations (chair, standing desk, etc).

I also suspect if a developer specifically asked for a desktop computer and could (lightly) justify the need for it, we would get them a desktop, and a laptop.

Re:Switching to AWS Graviton slashed our infrastructure bill... by throwaway821909 on Wednesday November 30, 2022 @04:36PM (#33808105)
Can you use a plain Hyper-V VM? i.e. with Hyper-V Manager
Re:Switching to AWS Graviton slashed our infrastructure bill... by gryf on Wednesday November 30, 2022 @04:38PM (#33808132)
Tried that but unfortunately there are some painful addressing and routing issues when you are subjected to when dealign with a corporate always on VPN. Ergo you can't actually contact clusters which you have to admin via kubectl.
Re:Switching to AWS Graviton slashed our infrastructure bill... by xenophonf on Wednesday November 30, 2022 @08:14PM (#33810865)
What's stopping you from running kubectl.exe or Cygwin? Frankly, I still think Cygwin's better than WSL in many ways.
Re:Switching to AWS Graviton slashed our infrastructure bill... by vetinari on Wednesday November 30, 2022 @04:53PM (#33808349)
FYI, with WSL2 1.0, you can finally enable systemd, so you can run services and cron jobs.
Re:Switching to AWS Graviton slashed our infrastructure bill... by gryf on Wednesday November 30, 2022 @04:55PM (#33808369)
Thanks for the tip off. I will look into this tomorrow. This is why I'm here. The distributed consciousness of HN is a wonderful problem solving engine :)
Re:Switching to AWS Graviton slashed our infrastructure bill... by dserodio on Wednesday November 30, 2022 @05:07PM (#33808528)
Never user Docker for Windows, but Docker for Mac has not been great lately either.

Granted, macOS changes a lot between each release but our company is paying for Docker Desktop licenses and the experience has really been disappointing.

Re:Switching to AWS Graviton slashed our infrastructure bill... by scarface74 on Wednesday November 30, 2022 @05:07PM (#33808535)
> Note to apple: please start concentrating on the enterprise sector. We're dying over here.

Enterprise sales are where the customer is not the user. Apple does best when the user is the purchaser.

Also, I know for a fact that Macs are well supported at scale by many large tech companies including my own.

Re:Switching to AWS Graviton slashed our infrastructure bill... by aidos on Wednesday November 30, 2022 @04:11PM (#33807748)
That’s the transition we went through. Our dependencies are / were pretty weird so the transition took a bit of effort - I suspect more complicated than many people would have to go through.

We all use Macs at work so we knew it was a matter of time before we were on ARM. I’m glad we made the transition. M1 airs are a delight to work with and Graviton machines are great bang for buck.

Re:Switching to AWS Graviton slashed our infrastructure bill... by wizofaus on Thursday December 01, 2022 @12:39AM (#33812771)
Valid why? Do you not trust compilers? Is it infeasible to (at least occasionally) run automated tests on cloud instances? Personally I've been pretty used to quite significant differences between local and production environments - it's rarely an issue, and I don't remember CPU architecture every being one. Things like timezones or firewall restrictions/ network differences (including talking to 3rd party APIs with IP whitelisting) are far more likely to cause problems.
Re:Switching to AWS Graviton slashed our infrastructure bill... by andrewstuart on Wednesday November 30, 2022 @03:31PM (#33807202)
If I started to build today I'd build and host my own servers, or go with servers from ionos. Cloud is very expensive.
Re:Switching to AWS Graviton slashed our infrastructure bill... by gryf on Wednesday November 30, 2022 @04:07PM (#33807703)
I just worked on a massive "optimised" cloud migration like you've never seen. We moved from multiple DCs to AWS and the costs are approximately 8x what the pre-migration costs are. We were realistically expecting 2x which gave us some regional agility and was expected but the unconstrained growth and misunderstanding of the cost model was terrible. It's designed to be so convoluted that you can't possibly estimate costs until you get the first bill at which point you are committed on a multi-month or year project. On top of that the assumption at the time of development is the cost is someone else's so the sprawl since the migration is dangerous which means we cannot leave ever now we've embraced the PaaS options.

The whole proposition relies on the idea of a sunk cost being accepted.

So yes, back to servers please. IaaS should be the maximal offering that is accepted by a business from a risk perspective unless the tool or technology is disposable in a 6 month window. There is space there for gains. PaaS hell no.

Edit: worth mentioning that AWS support is somewhere near dire. We've had issues with multiple services and despite being a VERY high roller with enterprise support we can't get anything fixed in any reasonable time. It's just someone else's crap you're using and they aren't any better at it than you are, just adding lead time to any issues. In some cases I've had to actually call out complete bad implementations that break function guarantees provided by open source projects (I can't logically warn people away from services as it's pretty obvious who I am if I do). One rule I've developed is that if it's not a core project: S3, EC2, EBS, ALB etc then it's probably a commercial liability in some way. There are no people working or with any knowledge on some major bits of AWS infra.

Re:Switching to AWS Graviton slashed our infrastructure bill... by tester756 on Wednesday November 30, 2022 @04:30PM (#33808018)
>We moved from multiple DCs to AWS and the costs are approximately 8x what the pre-migration costs are.

8x?? that's crazy, what where you doing wrong then?

Re:Switching to AWS Graviton slashed our infrastructure bill... by gryf on Wednesday November 30, 2022 @04:31PM (#33808036)
Everything, all at once.

SMEs can't reliably manage that transition with any skillset and still deliver a product at the same time.

Re:Switching to AWS Graviton slashed our infrastructure bill... by jiggawatts on Wednesday November 30, 2022 @05:07PM (#33808532)
I’m very curious to hear more details!

Did you use reservations to reduce costs?

Was it a lift and shift with VM configs staying as-is? (I’ve seen a lot of empty 1TB “app” drives burning money in the cloud!)

You complain about PaaS services, but I can’t imagine 8 data centres worth of stuff being converted to PaaS in hurry!

Re:Switching to AWS Graviton slashed our infrastructure bill... by gryf on Wednesday November 30, 2022 @05:20PM (#33808710)
Cost savings are mostly consolidation, scaling down stuff we don't need (we have peak hours) and migrating stuff to kubernetes and packing it tight.
Re:Switching to AWS Graviton slashed our infrastructure bill... by jiggawatts on Wednesday November 30, 2022 @05:35PM (#33808930)
Despite that you were paying 8x the previous amount!?

How is that possible?

E.g.: With Kubernetes and AWS you ought to be able to use clusters with a base of "reserved" capacity plus spot pricing for peak hours on top, right?

From what I've seen (in my limited experience), that should reduce costs for most orgs, not increase them!

Re:Switching to AWS Graviton slashed our infrastructure bill... by gryf on Wednesday November 30, 2022 @05:40PM (#33808991)
I have identified the limits of human incompetence.

Please someone hire me so I don't have to live through this nightmare any longer.

Re:Switching to AWS Graviton slashed our infrastructure bill... by yjftsjthsd-h on Wednesday November 30, 2022 @10:53PM (#33812113)
> How is that possible?

TLDR: AWS is really expensive compared the equivalent compute elsewhere, to the point of overwhelming its advantages.

---

Long answer that I wrote before realizing how long it was:

I'm not the person you're replying to, I don't know the exact details, I don't know their company. However:) I can tell you that AWS is a factor more expensive than renting or owning your own metal. They (claim to) offset this by 1. taking care of management for you so you need fewer ops people, and 2. letting you scale up and down as needed. The first is plausibly legitimate, but really depends on the size of your compute and the size of your human teams (and how much benefit you get from AWS-managed offerings). The second can help, but only if you've got really spiky workloads, that are only really running a tiny fraction of the time, with absolutely tiny base load. Scaling up and down helps reduce your AWS bill compared to not scaling up and down on AWS, of course, but it doesn't help that much when their elastic offerings are that much more expensive. Say, for the sake of argument that you can run most of your EC2 instances just 6 hours a day, during peak hours. That lets you win... if, and only if, EC2 instances are less than 4x the cost of just owning your own bare metal machines. I've gone so far as trying to price out using spot instances for CI work - just selectively, when they were the cheapest! - to augment instances on Hetzner Cloud. Guess what? They're so expensive that even at spot prices EC2 is a factor more expensive.

Re:Switching to AWS Graviton slashed our infrastructure bill... by jiggawatts on Thursday December 01, 2022 @02:10AM (#33813261)
Really?

I just priced up a 128-core (dual EYPC) Dell server with specs comparable to a matching Azure cloud server (HB120rs_v3), and the "1 year reserved" price for Azure came out to about the same as the purchase price over 3 years. The 3-year reserved price is about the same as the purchase price over 4 years. There's a new 5-year reservation option, which is the equivalent of owning the hardware for 7 years. Spot pricing is the equivalent of amortizing the purchase price over 12 years!!

Meanwhile, using the cloud, you can upgrade to the 9004 series in just a few months, not 12 years from now. And then whatever comes after the 9004 series in like 2 years, not 5, 7, or 12.

So it looks like that at the larger scales, the pricing is very directly competitive with on-premises hardware.

Consider that the cloud hosts include most basic operations costs, such as cooling, electricity, data centre floor space rental, the SFP ports and cabling, etc, etc...

Speaking of which, a quick back-of-the-envelope calculation is that a 128 core server will cost about $20K in cooling and electricity over its lifetime.

Note that the Azure HB120rs_v3 sizes come with 200 Gbps InfiniBand "just thrown in" for laughs. Try pricing that up some time, in case you want to build your own hyperconverged infrastructure!

Admittedly, at the smaller sizes, Azure and AWS are less competitive, but you do get flexibility, automation, and a bunch of other stuff that's difficult and expensive at scale on-prem.

Re:Switching to AWS Graviton slashed our infrastructure bill... by tester756 on Wednesday November 30, 2022 @05:22PM (#33808727)
it feels like the "refactor" was the way to improve
Re:Switching to AWS Graviton slashed our infrastructure bill... by gryf on Wednesday November 30, 2022 @05:23PM (#33808747)
I love reading people saying this. Try a couple of thousand DB tables evolved organically. Nope not happening.
Re:Switching to AWS Graviton slashed our infrastructure bill... by tester756 on Wednesday November 30, 2022 @05:45PM (#33809059)
I meant that what you did was "refactor" - rethinking the whole mess

So it wasn't purely cloud/renting racks issue

Re:Switching to AWS Graviton slashed our infrastructure bill... by Spivak on Wednesday November 30, 2022 @04:12PM (#33807771)
Rent servers: yes. Host your own: maybe. You can run a whole-ass company on 2 $70/mo servers from Hetzner (and some B2 for durable storage) while you figure out whether you have a market or not.

Like there's just no point in coloing when you're small because either all non-server bits will cost you for no reason or you're using something managed which is just cloud but more annoying.

Re:Switching to AWS Graviton slashed our infrastructure bill... by rjh29 on Wednesday November 30, 2022 @03:31PM (#33807206)
We are building everything for arm and I've surprised if other large companies aren't optimising for it.
Re:Switching to AWS Graviton slashed our infrastructure bill... by hdjjhhvvhga on Wednesday November 30, 2022 @03:28PM (#33807159)
If I started to build today, I'd definitely go for Hetzner Cloud. There is zero possibility that I get surprised by a large bill.
Re:Switching to AWS Graviton slashed our infrastructure bill... by jack_pp on Wednesday November 30, 2022 @03:30PM (#33807192)
I feel it's currently in beta, I've tried it and apparently I can't create more than a few instances because my account is "too new", without a clear way to remove that limit so you're right, can't have a large bill if you can't even create 10 instances.
Re:Switching to AWS Graviton slashed our infrastructure bill... by capableweb on Wednesday November 30, 2022 @03:37PM (#33807291)
Did you try writing them and asking them to increase the limit?

No cloud provider will give you the option to create as many instances as there are available ones, they all have limits from the get-go. Usually you have to write them/fill out some form if you want to go above the standard limit, Hetzner Cloud as well.

Re:Switching to AWS Graviton slashed our infrastructure bill... by jack_pp on Wednesday November 30, 2022 @04:10PM (#33807744)
I haven't, mainly because of this warning on their Limits page:

> Your account is too new to request a limit increase. Please note that we generally do not answer questions regarding limit increase on the telephone.

Re:Switching to AWS Graviton slashed our infrastructure bill... by hdjjhhvvhga on Thursday December 01, 2022 @02:46AM (#33813479)
It's very easy - you just write to them and ask to increase the limit to whatever you need. You need to do so in writing, not on the phone.
Re:Switching to AWS Graviton slashed our infrastructure bill... by Reitet00 on Wednesday November 30, 2022 @03:33PM (#33807227)
Too bad Hetzner doesn't have cheaper Arm servers available. Their Ampere pricing is not really convincing.
Re:Switching to AWS Graviton slashed our infrastructure bill... by yjftsjthsd-h on Wednesday November 30, 2022 @10:53PM (#33812118)
Who cares? Their x86 servers are cheap enough to not care about ARM.
Re:Switching to AWS Graviton slashed our infrastructure bill... by gryf on Wednesday November 30, 2022 @03:50PM (#33807465)
This. We have three people entirely dedicated to reducing costs.

As for avoiding ARM, we do only x86-64 because corporate security policy demands that we have Windows laptops so that some box ticking overlord can fill out a security policy compliance form. That means we're stuck limping along with docker and WSL2. Every single engineer in the org has an arm64 machine at home already and wants a proper computer at work, which can ironically work in the same policy framework if anyone gave enough of a shit to deal with it.

So that's why we don't use Graviton; corporate security policies. Our customers will just have to eat the price hikes.

Re:Switching to AWS Graviton slashed our infrastructure bill... by jiggawatts on Wednesday November 30, 2022 @06:09PM (#33809338)
You can cross-compile to ARM using Docker on Windows.

See: https://www.docker.com/blog/multi-arch-images/

Re:Switching to AWS Graviton slashed our infrastructure bill... by allset_ on Thursday December 01, 2022 @12:49AM (#33812835)
Builds in production shouldn't be built using developer laptops. I think you're approaching this wrong. You can build and test on x86_64 laptops all day if you want and still easily deploy to arm64 servers.
Re:Switching to AWS Graviton slashed our infrastructure bill... by gryf on Thursday December 01, 2022 @04:03AM (#33813977)
We don't do production builds on laptops.

But it's important that all builds are 100% reproducible on all build targets and that includes non docker ones. That is much more difficult if you have to cross compile stuff. We can barely manage one architecture.

Re:Switching to AWS Graviton slashed our infrastructure bill... by mk89 on Wednesday November 30, 2022 @03:58PM (#33807577)
I agree. Until it becomes an issue, where everyone runs screaming like chickens, literally nobody gives a shit.
Re:Switching to AWS Graviton slashed our infrastructure bill... by Thaxll on Wednesday November 30, 2022 @04:05PM (#33807675)
Because ARM perf was far far from being on part with Intel / AMD, also you need to be able to compile on that arch.
Re:Switching to AWS Graviton slashed our infrastructure bill... by nodesocket on Wednesday November 30, 2022 @04:07PM (#33807698)
Agree, I've gone to Graviton instances by default for RDS and ElastiCache (run and own a DevOps consulting company). The big problem that I continue to deal with is native arm64 Docker containers (if you a cool kid running containers / Kubernetes). For example, the very popular Bitnami charts don't support arm builds even though the community has been screaming for support.
Re:Switching to AWS Graviton slashed our infrastructure bill... by arecurrence on Wednesday November 30, 2022 @04:14PM (#33807806)
Completely agree... the only exception I've run into is that for small operations build tooling often doesn't work well with arm64.

EG: GitHub actions can build a container in a few minutes in x64 or 35 minutes in arm64... likewise aws-cdk literally could not run an arm64 fargate ecs deployment for months after support was added (They simply did not support the required attribute in the container definition).

I would love to see this change as I've had nothing but great experiences with graviton for virtually anything arm supported.

Re:Switching to AWS Graviton slashed our infrastructure bill... by _joel on Wednesday November 30, 2022 @04:58PM (#33808429)
Are you building on arm64 natively or via qemu. A few mins vs 35 for the same roughish spec of CPU, seems a bit off, even with optimisation considerations.

I've found arm64 builds on amd64 take longer when using one build context/arch (but doing multiple platforms), but that's as it's being emulated.

It's the oppostite on my M1, the buildx amd64 takes longer.

Re:Switching to AWS Graviton slashed our infrastructure bill... by arecurrence on Wednesday November 30, 2022 @07:15PM (#33810184)
oh it was with buildx (which uses qemu) as GitHub Actions run in x64. I was showing a specific example of arm64 build tooling challenges small startups encounter (Github Actions lack of arm job runners in this case). My arm64 builds on arm64 architecture scream.
Re:Switching to AWS Graviton slashed our infrastructure bill... by _joel on Wednesday November 30, 2022 @09:26PM (#33811515)
You can do a self hosted runner on arm though, even on a Raspberry Pi (if inclined), so not sure it's prohibitively expensive to a startup, could even afford a spot graviton :D

I've got to say though, I've not experience that delta in build times, even emulated on my machine.

Re:Switching to AWS Graviton slashed our infrastructure bill... by sofixa on Wednesday November 30, 2022 @05:07PM (#33808534)
> GitHub actions can build a container in a few minutes in x64 or 35 minutes in arm64

What type of container, and on what runner? That has not been my experience at all, a cross-compiling buildx build with Python and a bunch of libraries takes only slightly longer for arm64 than it did for x86.

Re:Switching to AWS Graviton slashed our infrastructure bill... by waych on Wednesday November 30, 2022 @05:13PM (#33808603)
My favorite way to watch this slow down is to introduce some node workloads into the build workflow.
Re:Switching to AWS Graviton slashed our infrastructure bill... by forty on Wednesday November 30, 2022 @04:16PM (#33807828)
First graviton is not magic. We switched our main service, which is a nodejs monolith, and did not get any cost improvement (we had to add more instances to handle the same workload, which ended up being equivalent cost wise). There are certainly use cases when it's better, but it doesn't seem to be the only and obvious choice for all use cases.

Second our laptops and our CI are amd64 machines, and being able to run the same docker images in prod and locally is nice, and not having to build the image with qemu on the CI is also good.

I don't mind cloud-ARM, but there definitely are good reasons not to use it (which of course don't apply to everyone)

Re:Switching to AWS Graviton slashed our infrastructure bill... by adrr on Wednesday November 30, 2022 @04:22PM (#33807913)
Switching to Gravitron isn't an automatic cost savings. Everything is optimized for x86. It maybe cheaper, but significantly slower. We've been trying to migrate for the last year for both cost saving but also we switched to ARM based laptops.
Re:Switching to AWS Graviton slashed our infrastructure bill... by tester756 on Wednesday November 30, 2022 @04:27PM (#33807974)
Let's go even further - "Cloud cost is underrated"
Re:Switching to AWS Graviton slashed our infrastructure bill... by jiggawatts on Wednesday November 30, 2022 @04:59PM (#33808440)
At $dayjob I found an unused box in the cloud running an expensive database engine. It was idle for months, created to be used by a consultant on a project that had wound up. The consultant had quit his consultancy on top of this.

I was told under no uncertain terms not to even think of touching this VM because “the budget has been approved”.

I was shocked at the flagrant waste of money and assumed it was a one-off aberration.

Nope, for months afterwards I kept hearing the same refrain from manager after manager, from product owners and dev team leads.

“Don’t touch! We fought hard for this budget! You’ll take it from our cold dead hands!”

Eventually I soured on the whole idea of cloud cost optimisation a service for unmotivated third parties and gave up on the whole notion.

Re:Switching to AWS Graviton slashed our infrastructure bill... by yazaddaruvala on Wednesday November 30, 2022 @05:17PM (#33808666)
FWIW, in these situations you're better off proposing:

"I'm going to reuse this VM, to help our ... fleet scale better."

That way your management continues to use their allocated budget, and your real prod systems work slightly better (also will eventually require less additional $ to scale up - helping the company i.e. shareholders).

The thing to remember:

You would assume all middle management really manages are a top line and a bottom line. Numbers related to their KPIs/OKRs are roughly a top line, and numbers related to their resources (humans and cloud infra budget) are roughly their bottom line.

The reality: Middle management's resources (humans and cloud infra budget) are not their bottom line. Middle management gets rewarded (promoted) when they have "enough scope", scope has roughly always been defined by number of people (it now also includes things like cloud cost budget). As such middle management has to say "we need to do more with less", but they are promoted based on these numbers going up!

Is this reward structure in the best interest of companies (i.e. customers and/or shareholders)? No, neither. Is there a better system? Not yet. Is the reward structure created by middle management for middle management? Likely.

So in the meanwhile, if you don't want to become unmotivated, might as well work within the current reward structures.

Re:Switching to AWS Graviton slashed our infrastructure bill... by jiggawatts on Wednesday November 30, 2022 @06:06PM (#33809308)
> help our ... fleet scale better."

Heh, I tried that too! I found a lot of setups using old HDD disks that were at 100% of their IOPS limits and CPUs that were idle. When they were built, Azure didn't have Premium SSD, so that's forgivable. I offered to rearrange where the expenditure goes, such that they have newer and faster CPUs, Premium SSD, but fewer cores which means a reduction in licensing costs. Cost neutral while improving their performance and capacity.

That got a very firm "Nope! Nope! Nope!" from most (but not all) teams as well.

> As such middle management has to say "we need to do more with less", but they are promoted based on these numbers going up!

I came to the same conclusion. Many project managers or product owners introduced themselves in the first meeting by proudly proclaiming the huge size of their operation. I.e.: "The system I'm responsible for is a multi-million dollar project with a small army of developers!"

I've also noticed that there's a tendency to exponentially blow out complexity of what ought to be a trivial system for the same reason. Something that could be static HTML on an S3 bucket or Azure Storage Account turns into a microservices monstrosity draped across three clouds and four external SaaS services.

Those resumes don't pad themselves.

Re:Switching to AWS Graviton slashed our infrastructure bill... by joshspankit on Wednesday November 30, 2022 @08:28PM (#33811005)
It’s as though they interpreted David Graeber’s Bullshit Jobs as a managerial playbook
Re:Switching to AWS Graviton slashed our infrastructure bill... by GuB-42 on Wednesday November 30, 2022 @06:23PM (#33809510)
Clever managers will repurpose that box.

One of my managers was a champion at that, taking scraps from everywhere for undercover projects. One of the few who managed to get things done in a highly bureaucratic company. Also helped by the fact he is good at picking competent people.

Re:Switching to AWS Graviton slashed our infrastructure bill... by raffraffraff on Thursday December 01, 2022 @02:46AM (#33813481)
Imagine seeing your startup grow into a company where bureaucracy rewards department heads who waste money now to protect their budget so they can keep wasting it next year...
Re:Switching to AWS Graviton slashed our infrastructure bill... by yjftsjthsd-h on Wednesday November 30, 2022 @10:37PM (#33812036)
> In the industry there's a strong avoidance of ARM compute instances for no good reason.

Not no reason; it adds work and risks incompatibility. Now, that work might be relatively small, and most software these days is compatible with aarch64, but compared to amd64 (which is the de-facto standard, already supported by everything, the default without needing to set anything up) it's still something, and businesses are risk-averse.