News for hackers. Stuff that matters.
Re:Intel's make-or-break 18A process node debuts for data ce... by throwup238 on Tuesday March 03, 2026 @03:53PM (#47238812)
> even the internal talent to know whether they are hiring a good infrastructure engineer or not during the interview process.

This is really the core problem. Every time I’ve done the math on a sizable cloud vs on-prem deployment, there is so much money left on the table that the orgs can afford to pay FAANG-level salaries for several good SREs but never have we been able to find people to fill the roles or even know if we had found them.

The numbers are so much worse now with GPUs. The cost of reserved instances (let alone on-demand) for an 8x H100 pod even with NVIDIA Enterprise licenses included leaves tens of thousands per pod for the salary of employees managing it. Assuming one SREs can manage at least four racks the hardware pays for itself, if you can find even a single qualified person.

Re:Intel's make-or-break 18A process node debuts for data ce... by tgrowazay on Tuesday March 03, 2026 @04:23PM (#47239231)
Self-hosted 8xH100 is ~$250k, depreciated across three years => $80k/year, with power and cooling => $90k/year (~$10/hour total).

AWS charges $55/hour for EC2 p5.48xlarge instance, which goes down with 1 or 3 year commitments.

With 1 year commitment, it costs ~$30/hour => $262k per year.

3-year commitment brings price down to $24/hour => $210k per year.

This price does NOT include egress, and other fees.

So, yeah, there is a $120k-$175k difference that can pay for a full-time on-site SRE, even if you only need one 8xH100 server.

Numbers get better if you need more than one server like that.

Re:Intel's make-or-break 18A process node debuts for data ce... by Aurornis on Tuesday March 03, 2026 @04:33PM (#47239352)
$120K isn't going to cover the fully loaded costs of an SRE who can set up and run that.

Hiring 1 person to run the infrastructure means that 1 person is on-call 24/7 forever.

If there's an issue with the server while they're sick or on vacation, you just stop and wait.

If they take a new job, you need to find someone to take over or very quickly hire a replacement.

There's a second bus factor: What happens when that 8xH100 starts to get flakey? You can't move the jobs to another server because you only have one. You can start diagnosing things and replacing parts and hope it gets to the root issue, but that's more downtime.

Going on-prem like this is highly risky. It works well until the hardware starts developing problems or the person in charge gets a new job. The weeks and months lost to dealing with the server start to become a problem. The SRE team starts to get tired of having to do all of their work on weekends because they can't block active use during the week. Teams start complaining that they need to use cloud to keep their project moving forward.

Re:Intel's make-or-break 18A process node debuts for data ce... by Figs on Tuesday March 03, 2026 @04:52PM (#47239563)
> $120K isn't going to cover the fully loaded costs of an SRE who can set up and run that.

> Hiring 1 person to run the infrastructure means that 1 person is on-call 24/7 forever.

> If there's an issue with the server while they're sick or on vacation, you just stop and wait.

Very much depends on what you're doing, of course, but "you just stop and wait" for sickness/vacation sometimes is actually good enough uptime -- especially if it keeps costs down. I've had that role before... That said, it's usually better to have two or three people who know the systems though (even if they're not full time dedicated to them) to reduce the bus factor.

Re:Intel's make-or-break 18A process node debuts for data ce... by roryirvine on Wednesday March 04, 2026 @05:23AM (#47245515)
So the entire business was happy to go offline for 2/3 weeks whenever their infra person fancied going off on their summer holiday?

By doing this, you're guaranteeing a bus factor of below 1. I can't think of any business that wouldn't see that as being a completely unacceptable risk.

Re:Intel's make-or-break 18A process node debuts for data ce... by jononor on Wednesday March 04, 2026 @08:24AM (#47247074)
Bus factor 1 is rarely enough for "entire business". But if the GPUs are for training models, and their users are the data scientists that are also on holiday around the same times - that might indeed be good enough policy.
Re:Intel's make-or-break 18A process node debuts for data ce... by Aurornis on Wednesday March 04, 2026 @11:24AM (#47249816)
> and their users are the data scientists that are also on holiday around the same times

I’ve seen this before. It turns into restrictions on when you can schedule vacation times.

Not fun when your family wants to go on a trip but you can’t get the time off because it’s not one of the allowed vacation times.

Re:Intel's make-or-break 18A process node debuts for data ce... by jononor on Wednesday March 04, 2026 @02:17PM (#47252417)
Ouch, that is indeed a risk one must be wary of. Can be a "works for the company but sucks for employees". Which can also drain the company of skilled people, a poor trade in most cases.
Re:Intel's make-or-break 18A process node debuts for data ce... by Aurornis on Wednesday March 04, 2026 @11:23AM (#47249794)
I agree.

I never understand the drive to stay away from cloud services for small scale operations. It’s not your money that’s being spent on the cloud, but it is your free time being asked to be on call when you encourage your company to self-host!

Re:Intel's make-or-break 18A process node debuts for data ce... by Manuel_D on Tuesday March 03, 2026 @07:32PM (#47241319)
> There's a second bus factor: What happens when that 8xH100 starts to get flakey? You can't move the jobs to another server because you only have one.

You can still use cloud for excess capacity when needed. E.g. use on-prem for base load, and spin up cloud instances for peaks in load.

Re:Intel's make-or-break 18A process node debuts for data ce... by stego-tech on Wednesday March 04, 2026 @08:32AM (#47247158)
This is my favorite use of the public cloud: the modern-day “hot site”. It’s way cheaper to just pay reserved rates for failover instances of critical infra than a whole other unused site, assuming your particular compliance or regulatory frameworks allow it. Especially in an era of remote work, it’s highly practical and cost-effective.
Re:Intel's make-or-break 18A process node debuts for data ce... by PunchyHamster on Tuesday March 03, 2026 @06:29PM (#47240625)
> There's a second bus factor: What happens when that 8xH100 starts to get flakey? You can't move the jobs to another server because you only have one. You can start diagnosing things and replacing parts and hope it gets to the root issue, but that's more downtime.

they come with warranty, often with technican guaranteed to arrive within few hours or at most a day. Also if SHTF just getting cloud to augument current lackings isn't hard

Re:Intel's make-or-break 18A process node debuts for data ce... by justsomehnguy on Tuesday March 03, 2026 @05:12PM (#47239805)
If a business which require at least a quarter million bucks worth of hardware for the basic operation yet it can't pay the market rate for someonr who would operate it - maybe the basics of that business is not okay?
Re:Intel's make-or-break 18A process node debuts for data ce... by stego-tech on Wednesday March 04, 2026 @08:30AM (#47247137)
This.

Companies following consultant reports will usually end up offering 50% ranges, which for SRE/SIE roles in major metros comes to around $163k. If they study BLS/FRED/CPI data and aim to pay someone enough for a 50/30/20 budget in a major metro at median rent, they’ll offer $175k to $200k+. If they want someone to stick around, buy an average home, lay roots, it’s $210k+, minimum.

“Six figures” doesn’t cover essentials anymore for almost every major city in the USA, and the last thing you can afford to cheap out on is the labor supporting your IT infra. Every corner you cut today on TC (outsourcing, offshoring, consulting) is just letting fires rage until you either parachute out or everything burns down, and that’s not a game you can afford to play with critical business technologies.

Re:Intel's make-or-break 18A process node debuts for data ce... by Aurornis on Wednesday March 04, 2026 @11:27AM (#47249866)
I’m not disagreeing. I’m explaining to the commenter above that $120K isn’t going to cover the costs of a full-time SRE who will be on call 24/7

If a business can’t afford a properly staffed crew with enough allowance to cover a rotation of on call duties and allow for vacations, they should prefer the managed cloud services.

You’re paying more but you’re buying freedom and flexibility.

Re:Intel's make-or-break 18A process node debuts for data ce... by formerly_proven on Tuesday March 03, 2026 @04:39PM (#47239435)
> There's a second bus factor: What happens when that 8xH100 starts to get flakey?

These come in a non-flakey variant?

Re:Intel's make-or-break 18A process node debuts for data ce... by spwa4 on Tuesday March 03, 2026 @05:33PM (#47240052)
It's called a warranty.

And the other argument: every company I've ever know to do AWS has an AWS sysadmin (sorry "devops"), same for Azure. Even for small deployments. And departments want their own person/team.

Re:Intel's make-or-break 18A process node debuts for data ce... by Aurornis on Wednesday March 04, 2026 @11:28AM (#47249885)
You can tell in this thread who has and who hasn’t had to work with this hardware.

My favorite are the responses from people saying the warranty will have someone show up in “hours” and fix it. Best of luck to you.

Re:Intel's make-or-break 18A process node debuts for data ce... by charcircuit on Tuesday March 03, 2026 @06:07PM (#47240396)
>If there's an issue with the server while they're sick or on vacation, you just stop and wait.

You can ask AI to troubleshoot and fix the issue.

Re:Intel's make-or-break 18A process node debuts for data ce... by stego-tech on Tuesday March 03, 2026 @07:28PM (#47241279)
Out of all the comments on numbers, SREs, and scaling, you get the response for meeting numbers with numbers!

> $120K isn't going to cover the fully loaded costs of an SRE who can set up and run that.

Literally this. I can do SRE on-prem and cloud, and my 50/30/20 budget break-even point (as in, needs and savings but no wants - so 70%) is $170k before taxes. Rent is astonishingly high right now, and the sort of mid-career professional you want to handle SRE for your single DC is going to take $150k in this market before fucking off to the first $200k job they get.

Know your market, and pay accordingly. You cannot fuck around with SREs.

> Hiring 1 person to run the infrastructure means that 1 person is on-call 24/7 forever.

This is less of an issue than you might think, but strongly dependent upon the quality of talent you’ve retained and the budget you’ve given them. Shitbox hardware or cheap-ass talent means you’ll need to double or triple up locally, but a quality candidate with discretion can easily be supported by a counterpart at another office or site, at least short-term. Ideally though, yeah, you’ll need two engineers to manage this stack, but AWS savings on even a modest (~700 VMs) estate will cover their TC inside of six months, generally.

> There's a second bus factor: What happens when that 8xH100 starts to get flakey? You can't move the jobs to another server because you only have one. You can start diagnosing things and replacing parts and hope it gets to the root issue, but that's more downtime.

This strikes at another workload I neglected to mention, and one I highly recommend keeping in the public cloud: GPUs.

GPUs on-prem suck. Drivers are finnicky, firmware is flakey, vendor support inconsistent, and SR-IOV is a pain in the ass to manage at scale. They suck harder than HBAs, which I didn’t think was possible.

If you’re consuming GPUs 24x7 and can afford to support them on-prem, you’re definitely not here on HN killing time. For everyone else, tune your scaling controls on your cloud provider of choice to use what you need, when you need it, and accept the reality that hyperscalers are better suited for GPU workloads - for now.

> Going on-prem like this is highly risky.

Every transaction is risky, but the risk calculus for “static” (ADDS) or “stable” (ERP, HRIS, dev/test) work makes on-prem uniquely appealing when done right. Segment out your resources (resist the urge for HPC or HCI), build sensible redundancies (on-prem or in the cloud), and lean on workhorse products over newer, fancier platforms (bulletproof hypervisors instead of fragile K8s clusters), and you can make the move successful and sensible. The more cowboy you go with GPUs, K8s, or local Terraform, the more delicate your infra becomes on-prem - and thus the riskier it is to keep there.

Keep it simple, silly.

Re:Intel's make-or-break 18A process node debuts for data ce... by lazylizard on Tuesday March 03, 2026 @10:09PM (#47242514)
i am not sre, merely sysadmin.

and somehow i have this impression that gpus on slurm/pbs could not be simpler.

u can use a vm for the head node, dont even need the clustering really..if u can accept taking 20min to restore a vm.. and the rest of the hardware are homogeneous - you setup 1 right and the rest are identical.

and its a cluster with a job queue.. 1 node going down is not the end of the world..

ok if u have pcie GPUs sometimes u have to re-seat them and its a pain. otherwise if ur h200 or disks fail u just replace them, under warranty or not...

Re:Intel's make-or-break 18A process node debuts for data ce... by stego-tech on Wednesday March 04, 2026 @08:24AM (#47247070)
That sounds way easier than the methods I’ve had to manage GPUs in the Enterprise on-prem thus far (PCIe cards slotted into hypervisor boxes and shared via SR-IOV). I’ll have to look into it, but I doubt it’ll ever enter my personal wheelhouse given how quickly GPU-based workloads are either moved to the cloud for effective utilization at scale, or onto custom accelerators for edge workloads/inference.
Re:Intel's make-or-break 18A process node debuts for data ce... by throwup238 on Tuesday March 03, 2026 @10:43PM (#47242783)
> Out of all the comments on numbers, SREs, and scaling, you get the response for meeting numbers with numbers!

>> $120K isn't going to cover the fully loaded costs of an SRE who can set up and run that.

> Literally this. I can do SRE on-prem and cloud, and my 50/30/20 budget break-even point (as in, needs and savings but no wants - so 70%) is $170k before taxes. Rent is astonishingly high right now, and the sort of mid-career professional you want to handle SRE for your single DC is going to take $150k in this market before fucking off to the first $200k job they get.

That's $120k per pod. Four pods per rack at 50kW.

What universe are we living in that a single SRE can't manage even a single rack for less than half a million in total comp?

Re:Intel's make-or-break 18A process node debuts for data ce... by stego-tech on Wednesday March 04, 2026 @08:21AM (#47247041)
> What universe are we living in that a single SRE can't manage even a single rack for less than half a million in total comp?

The kind where TC isn’t measured by pod managed, but by person hired. Also the world where median rent in major metros is $3500 a month.

If you think $120k is rich, you’re either operating in the boonies, outside the USA/Canada, or incredibly out of touch with the cost of living today and need to seriously go study BLS/FRED/CPI data sets to understand how expensive it is to live right now.

Re:Intel's make-or-break 18A process node debuts for data ce... by ragall on Wednesday March 04, 2026 @11:26AM (#47249845)
> outside the USA/Canada

Indeed, there's no reason for a company to host this kind of batch compute in North America. You can get very good people in Eastern Europe at 1/3 the cost.

Re:Intel's make-or-break 18A process node debuts for data ce... by Aurornis on Wednesday March 04, 2026 @11:30AM (#47249928)
I like how this simple claim about being cheaper to self-host a single server has now escalated to opening an office in Eastern Europe and hiring people there to manage it.
Re:Intel's make-or-break 18A process node debuts for data ce... by ragall on Wednesday March 04, 2026 @12:10PM (#47250602)
The trend of opening offices in Europe started one year into Covid. I'm sure that there are companies that haven't opened an office there yet, but fewer than one might imagine.
Re:Intel's make-or-break 18A process node debuts for data ce... by everforward on Tuesday March 03, 2026 @05:50PM (#47240218)
I work in SRE and the way you describe it would give me pause.

The first is that SRE team size primarily scales with the number of applications and level of support. It does scale with hardware but sublinearly, where number of applications usually scales super linearly. It takes a ton less effort to manage 100 instances of a single app than 1 instance of 100 separate apps (presuming SRE has any support responsibilities for the app). Talking purely in terms of hardware would make me concerned that I’m looking at an impossible task.

The second (which you probably know, but interacts with my next point) is that you never have single person SRE teams because of oncall. Three is basically the minimum, four if you want to avoid oncall burnout.

The last is that I don’t know many SREs (maybe none at all) that are well-versed enough in all the hardware disciplines to manage a footprint the size we’re talking. If each SRE is 4 racks and a minimum team size is 4, that’s 16 racks. You’d need each SRE to be comfortable enough with networking, storage, operating system, compute scheduling (k8s, VMWare, etc) to manage each of those aspects for a 16 rack system. In reality, it’s probably 3 teams, each of them needs 4 members for oncall, so a floor of like 48 racks. Depending on how many applications you run on 48 racks, it might be more SREs that split into more specialized roles (a team for databases, a team for load balancers, etc).

Numbers obviously vary by level of application support. If support ends at the compute layer with not a ton of app-specific config/features, that’s fewer folks. If you want SRE to be able to trace why a particular endpoint is slow right now, that’s more folks.

Re:Intel's make-or-break 18A process node debuts for data ce... by PunchyHamster on Tuesday March 03, 2026 @06:27PM (#47240607)
> The last is that I don’t know many SREs (maybe none at all) that are well-versed enough in all the hardware disciplines to manage a footprint the size we’re talking. If each SRE is 4 racks and a minimum team size is 4, that’s 16 racks. You’d need each SRE to be comfortable enough with networking, storage, operating system, compute scheduling (k8s, VMWare, etc) to manage each of those aspects for a 16 rack system. In reality, it’s probably 3 teams, each of them needs 4 members for oncall, so a floor of like 48 racks. Depending on how many applications you run on 48 racks, it might be more SREs that split into more specialized roles (a team for databases, a team for load balancers, etc).

That's vastly overstating it. You hit nail in the head in previous paragraphs, it's number of apps (or more generally speaking ,environments) that you manage, everything else is secondary.

And that is especially true with modern automation tools. Doubling rack count is big chunk of initial time spent moving hardware of course, but after that there is almost no difference in time spent maintaining them.

In general time per server spent will be smaller because the bigger you grow the more automation you will generally use and some tasks can be grouped together better.

Like, at previous job, server was installed manually, coz it was rare.

At my current job it's just "boot from network, pick the install option, enter the hostname, press enter". Doing whole rack (re)install would take you maybe an hour, everything else in install is automated, you write manifest for one type/role once, test it, and then it doesn't matter whether its' 2 or 20 servers.

If we grew server fleet say 5-fold, we'd hire... one extra person to a team of 3. If number of different application went 5-fold we'd probably had to triple the team size - because there is still some things that can be made more streamlined.

Tasks like "go replace failed drive" might be more common but we usually do it once a week (enough redundancy) for all servers that might've died, if we had 5x the number of servers the time would be nearly the same because getting there dominates the 30s that is needed to replace one.

Re:Intel's make-or-break 18A process node debuts for data ce... by ori_b on Tuesday March 03, 2026 @06:40PM (#47240761)
Noteworthy: the number of apps isn't affected by whether the machines are in your datacenter or Amazon's.
Re:Intel's make-or-break 18A process node debuts for data ce... by esseph on Tuesday March 03, 2026 @06:34PM (#47240702)
So your definition of SRE is anybody that works on infra?
Re:Intel's make-or-break 18A process node debuts for data ce... by everforward on Wednesday March 04, 2026 @01:12PM (#47251473)
I would call what you’re describing Datacenter Operations, with the exception of PXE boot.

You could have SRE do it, but most places don’t because you can get someone to swap a dead drive for way cheaper (it’s not really a complicated operation).

That growth of SRE teams comes from wanting reliability further up the stack. If you’re not on AWS, there’s no Aurora so someone has to be DBA to do backups, performance monitoring, configuring failovers for when a disk dies and RAID needs to rebuild, etc. Same for network, networked storage, yada yada

Re:Intel's make-or-break 18A process node debuts for data ce... by skissane on Tuesday March 03, 2026 @10:09PM (#47242511)
> The first is that SRE team size primarily scales with the number of applications and level of support. It does scale with hardware but sublinearly, where number of applications usually scales super linearly. It takes a ton less effort to manage 100 instances of a single app than 1 instance of 100 separate apps (presuming SRE has any support responsibilities for the app). Talking purely in terms of hardware would make me concerned that I’m looking at an impossible task.

Never been an SRE but interact with them all the time…

My own personal experience is there is commonly a division between App SREs that look after the app layer and Infra SREs that looks after the infrastructure layer (K8S, storage, network, etc)

The App SRE role absolutely scales with the number of distinct apps. The extent to which the Infra SRE role does depends on how diverse the apps are in terms of their infrastructure demands

Re:Intel's make-or-break 18A process node debuts for data ce... by everforward on Wednesday March 04, 2026 @01:18PM (#47251569)
Yeah, that’s valid, there are a few common layouts for SRE. I would call what you’re describing a horizontal layout (each team owns a layer for all apps that use that layer).

It sort of comes back to support levels. Your Infra SRE teams stay small if either a) an app SRE team owns application specific stuff, or b) SRE just doesn’t support application specific stuff. Eg if a particular query is slow but the DB is normal, who owns root causing that? Whoever does needs headcount, whether it’s app SRE, infra SRE or the devs.

Re:Intel's make-or-break 18A process node debuts for data ce... by kaliszad on Wednesday March 04, 2026 @09:06AM (#47247533)
Many people assume that companies need or want global enterprise level of management of infrastructure or 24/7 support. That's simply not the case. Many small and mid-sized companies just need their applications to run. There is no CTO on the board and nobody else really cares where the stuff runs if it fits a certain budget, is available enough to not cause major disruptions and is responsive enough to not cause complaints. Some companies may care about a certain level of compliance/ security and whether their admins/ DevOps people seem to be in agony most of the time but of those there aren't many. That's also a reason why the EU introduced directives such as NIS2, DORA, CRA, CER, even the now 10 year old GDPR and more.

Most companies I have seen have never updated the BIOS of their servers, nor the firmware on their switches. Some of those have production applications on Windows XP or older and you can see VMware ESXi < 6.5 still in the wild. The same for all kinds of other systems including Oracle Linux 5.5 with some ancient Oracle DB like 10g or something, that was the case like 5 years ago but I don't think the company has migrated away completely to this day.

Any sufficiently old company will accrete systems and approaches of various vintages over time only very slowly ripping out some of those systems. Usually what happens is that parts of old systems or old workarounds will live on for decades after they have been supposedly decommissioned. I had a colleague who was using CRT monitors in 2020 with computers of similar vintage, probably with Pentium III or early Pentium IV, because he had everything set up there and it just worked for what he was doing. I don't admire it, yet that stuff works and I do respect that people don't want to replace expensive systems just because they are out of support, when they do actually work and they have people taking care of them.

Re:Intel's make-or-break 18A process node debuts for data ce... by everforward on Wednesday March 04, 2026 @01:24PM (#47251661)
Totally, but then you probably don’t want SREs. If you’re okay with 99% availability (~7 hours of downtime a month assuming 24x7 goal), you can get by with much cheaper staffing and won’t have to deal with the turnover from SREs who get bored.
Re:Intel's make-or-break 18A process node debuts for data ce... by Eridrus on Tuesday March 03, 2026 @06:13PM (#47240470)
I disagree with on-prem being ideal for GPU for most people.

If you're doing regular inference for a product with very flat throughput requirements (and you're doing on-prem already), on-prem GPUs can make a lot of sense.

But if you're doing a lot of training, you have very bursty requirements. And the H100s are specifically for training.

If you can have your H100 fleet <38% utilized across time, you're losing money.

If you have batch throughput you can run on the H100s when you're not training, you're probably closer to being able to wanting on-prem.

But the other thing to keep in mind is that AWS is not the only provider. It is a particularly expensive provider, and you can buy capacity from other neoclouds if you are cost-sensitive.

Re:Intel's make-or-break 18A process node debuts for data ce... by harrall on Tuesday March 03, 2026 @08:42PM (#47241906)
You didn’t find people because SREs don’t do that.

You wanted sysadmins / IT / data center technicians.

Re:Intel's make-or-break 18A process node debuts for data ce... by red-iron-pine on Wednesday March 04, 2026 @09:01AM (#47247486)
yeah homie is talking about DevSecOps and what he needs to hire is a cable monkey

no shortage of IT talent in 2026, the market is literally overflowing with resumes and wages are dropping. huge gluts of fairly generic online degree holders.

they can use AI to write basic Ansible just as well as my Seniors