News for hackers. Stuff that matters.
Re:Oxide: The Cloud Computer by Always_Anon on Thursday October 26, 2023 @02:52PM (#38030123)
>Sure you can, but then who will diagnose and fix your hardware/OS interaction problems when you have parts from five vendors in the mix?

Dell is a single vendor that will diagnose and fix all of your hardware issues.

With Oxide you're locked into what looks like a Solaris derivative OS running on the metal and you're only allowed to provision VMs which is a huge disadvantage.

I run a fleet of over 30,000 nodes in three continents and the majority is Flatcar Linux running on bare metal. Also have a decent amount of RHEL running for specific apps. We can pick and choose our bare metal OS which is something you cannot do with Oxide. That's a tough pill to swallow.

Re:Oxide: The Cloud Computer by wmf on Thursday October 26, 2023 @03:09PM (#38030351)
Dell is a single vendor that will diagnose and fix all of your hardware issues.

And you'll be down for weeks or months while they do it.

Re:Oxide: The Cloud Computer by Always_Anon on Thursday October 26, 2023 @03:17PM (#38030442)
>And you'll be down for weeks or months while they do it.

Off by a few orders of magnitude. Dell on-site SLA with pre-purchased spares was about 6 hours.

With Oxide, you'd be lucky to get same day service.

Re:Oxide: The Cloud Computer by jjav on Thursday October 26, 2023 @04:02PM (#38030991)
> Off by a few orders of magnitude. Dell on-site SLA with pre-purchased spares was about 6 hours.

You're talking about replacement parts. Yes Dell is good about that.

The discussion above is asking them to diagnose and fix a problem with the interaction of various hardware components (all of which come from third parties).

Re:Oxide: The Cloud Computer by Always_Anon on Thursday October 26, 2023 @04:13PM (#38031118)
Oxide also has various hardware components from AMD, Intel, Samsung, etc. They are not manufacturing every component.
Re:Oxide: The Cloud Computer by mlindner on Friday October 27, 2023 @01:23AM (#38034995)
But they _are_ writing the firmware that runs most of them and need to understand those devices at a deep level in order to do that, unlike Dell. Dell slaps together hardware and firmware from other vendors with some high level software of their own on top. They don't do the low level firmware and thus don't understand the low level intricacies of their own systems.
Re:Oxide: The Cloud Computer by Always_Anon on Friday October 27, 2023 @01:45AM (#38035100)
No they're not unless I'm mistaken. They're not writing the firmware that runs on the NVMe drives, nor the NICs (they're not even writing the drivers for some of the NICs), etc.

There's a line in the sand that you must cross when it comes to understanding the true nature of the componentry that you're using. At the end of the day, your AMD CPUs may be lying to you, to all of us, but we just don't know it yet.

Re:Oxide: The Cloud Computer by wmf on Thursday October 26, 2023 @04:05PM (#38031028)
I'm not speaking hypothetically. If you hit a "zero-day" bug that Dell has never seen it's going to take time. And somehow every large customer finds bugs that Dell certification didn't.
Re:Oxide: The Cloud Computer by Always_Anon on Thursday October 26, 2023 @04:12PM (#38031094)
>I'm not speaking hypothetically.

Neither am I.

>If you hit a "zero-day" bug that Dell has never seen it's going to take time.

If you hit a "zero-day" bug that Oxide has never seen it's going to take time.

>And somehow every large customer finds bugs that Dell certification didn't.

Yes, happens. And I'm sure the exact same will happen with Oxide, so it's not a differentiator.

Re:Oxide: The Cloud Computer by pests on Thursday October 26, 2023 @05:58PM (#38032239)
> And somehow every large customer finds bugs that Dell certification didn't.

It's a law of computer engineering.

In the Apollo 11 decent sequence the Rendezvous Radar experienced a hardware bug[0] not uncovered during simulation. They found it later, but until then, the solution was adding a "turn off Rendezvous Radar" checklist item.

[0] The Rendezvous Radar would stop the CPU, shuttle some data into areas it could be read, and woke the CPU back up to process it. The bug caused it to supuriously do this dance just to tell it "no new data", which then caused other systems to overload.

Re:Oxide: The Cloud Computer by jjav on Thursday October 26, 2023 @04:06PM (#38031039)
> Dell is a single vendor that will diagnose and fix all of your hardware issues.

I've been a Dell customer at a previous company. I know for a fact that's not true.

I had a support ticket for a weird firmware bug open for two years, they could never figure it out. I left that job but for all I know the case is still open many years later.

Dell doesn't know how to fix things like that because they don't design and engineer the systems they sell. Dell is a reseller who puts components together from a bunch of vendors and it mostly works but when it doesn't, there's nobody on staff who can fix it.

Re:Oxide: The Cloud Computer by Always_Anon on Thursday October 26, 2023 @04:28PM (#38031304)
I've been a Dell customer for decades at this rate and I know for a fact it's true.

I've had support tickets open for all kinda of weird firmware, hardware, etc. bugs and they've been well resolved, even if it meant Dell just replaced the part with something comparable (NIC swap).

>Dell doesn't know how to fix things like that because they don't design and engineer the systems they sell.

Of course they do. That's like saying Oxide doesn't know how to fix stuff because they don't design the CPU, NVMe, DIMMs, etc. Oxide is still going to vendors for these things.

Re:Oxide: The Cloud Computer by bcantrill on Thursday October 26, 2023 @10:51PM (#38034145)
Ironically, it was Dell's total inability to resolve a pathological rash of uncorrectable memory errors very much is part of the origin story of Oxide: this issue was very important to my employer (who was a galactic Dell customer) and as the issue endured and Dell escalated internally, it became increasingly clear that there was in fact no one at Dell who could help us -- Dell did not understand how their own systems work.

At Oxide, we have been deliberate at every step, designing from first principles whenever possible. (We -- unlike essentially everyone else -- did not simply iterate from a reference design.)

To make this concrete with respect to the CPU in particular, we have done our own lowest-level platform enablement software[0] -- we have no BIOS. No one -- not the hyperscalers, not the ODMs and certainly not Dell -- has done this, and even AMD didn't think we could pull it off. Why did we do it this way? Because all along our lodestar was that problem that Dell was useless to us on -- that we wanted to understand these systems from first principles, because we have felt that that is essential to deliver the product that we ourselves wanted to by.

There are plenty of valid criticisms of Oxide -- but that we don't understand our system simply isn't one of them.

[0] https://www.osfc.io/2022/talks/i-have-come-to-bury-the-bios-...

Re:Oxide: The Cloud Computer by mlindner on Friday October 27, 2023 @01:15AM (#38034952)
As a side question, what's the name of your custom firmware that is the replacement of the AGESA bootloader? I tried searching on the oxide github page but couldn't find anything that seemed to fit that description.
Re:Oxide: The Cloud Computer by bcantrill on Friday October 27, 2023 @02:26AM (#38035304)
(The AGESA bootloader -- or ABL -- is in the AMD PSP.) In terms of our replacement for AGESA: the PSP boots to our first instruction, which is the pico host bootloader, phbl[0]. phbl then loads the actual operating system[1], which performs platform enablement as part of booting. (This is pretty involved, but to give you a flavor, see, e.g. initialization of the DXIO engine.[2])

[0] https://github.com/oxidecomputer/phbl

[1] https://github.com/oxidecomputer/illumos-gate/tree/stlouis

[2] https://github.com/oxidecomputer/illumos-gate/blob/stlouis/u...

Re:Oxide: The Cloud Computer by mlindner on Friday October 27, 2023 @03:03AM (#38035537)
Thanks, are the important oxide branches of illumos-gate repo (and any other cloned repos) defined anywhere? I definitely wouldn't have found that branch without you mentioning it here.
Re:Oxide: The Cloud Computer by Always_Anon on Friday October 27, 2023 @01:37AM (#38035062)
Interesting enough I also ran into something somewhat related with Dell that they were not able to resolve so they ended up working in a replacement from another vendor.

Nonetheless, it is quite interesting what you've built, but as the end user I'm not quote convinced that it matters. Sure you can claim it reduces attack vectors and such but we'll still see Dells and IBMs in the most restricted and highest security postured sites in the world. Think DoD and such.

The software management plane y'all built is the headlining feature IMHO, not so much what happens behind the scenes that the vast majority of the time will not have a fatal catastrophic upstream effect.

>There are plenty of valid criticisms of Oxide -- but that we don't understand our system simply isn't one of them.

That's not what I said. There's a line in the sand that you must cross when it comes to understanding the true nature of the componentry that you're using. At the end of the day, your AMD CPUs may be lying to you, to all of us, but we just don't know it yet.

Re:Oxide: The Cloud Computer by samcat116 on Thursday October 26, 2023 @04:28PM (#38031302)
The vast majority of people only need to deploy VMs.
Re:Oxide: The Cloud Computer by Always_Anon on Thursday October 26, 2023 @04:30PM (#38031321)
It's ironic coming from a company who's CTO has harped about containers on bare metal for years. Maybe a large swath only need to deploy VMs, but the future will most definitely involve bare metal for many use cases, and oddly Oxide doesn't support that currently.
Re:Oxide: The Cloud Computer by pseg134 on Thursday October 26, 2023 @05:45PM (#38032101)
I run a battalion of 78,000 nodes and I disagree with you.
Re:Oxide: The Cloud Computer by Always_Anon on Thursday October 26, 2023 @06:01PM (#38032269)
I used to run over 150,000 nodes and I agree with me.
Re:Oxide: The Cloud Computer by jryle70 on Thursday October 26, 2023 @10:23PM (#38033964)
See the pattern? Dell only care about the big guys.

Set aside the childish tone ...

> Dell is a single vendor that will diagnose and fix all of your hardware issues.

There are two anecdotes here disagreeing with you, and frankly that's enough to say what you said above isn't true, not universally so. I doubt Odixe is targeting big deployment like yours, but more like theirs. Whether they will succeed is another matter, but they do have a valid sales pitch and the expertise to pull it off.